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ABSTRACT 

This document considers both self-examination and 
external evaluation of gifted and talented education programs. 
Principles of the self-examination process are offered, noting 
similarities to external evaluation models . Principles of 
self-evaluation efforts include the importance of maintaining a 
nonjudgmental orientation, soliciting views from the insider's 
perspective, and triangulating data. The most important techniques in 
self-evaluation are observation, interviewing, and participation.' 
While self-evaluation maintains an educational program ! s quality on a 
daily basis, expert external evaluation is essential to an in-depth 
and objective understanding. Important considerations in external 
evaluation include working with an external evaluator and 
understanding qualitative and quantitative approaches. Clear 
communi cat i on of evaluat i on findings i s important for both internal 
and external evaluations. General evaluation guidelines and specific 
guidelines for evaluating gifted and talented education programs are 
given. A case study illustrates many of these guidelines, including 
the importance of context; preparing a program description; the use 
of verbatim quotations; assessment and analysis; review of the 
standard program mechanisms (referral, identification, and 
selection); refinements; analyzing underlying factors; addressing 
larger sociopolitical concerns ; and communicating evaluation 
findings. Appendices provide further explanation of evaluation 
components . (Contains 42 references . ) (DB) 
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ABSTRACT 

The health of a gifted and talented program requires both self-examination and 
external evaluation. Routine self-examination allows early detection of educational 
problems and confirmation of a sound programmatic approach. By detecting and thus 
helping to prevent program deterioration, self-examination — together with its corollary, 
external evaluation— promotes the intellectual and artistic health of gifted and talented 
education programs. This discussion will highlight some of the common sense ways of 
reflecting upon one's programmatic achievements and shortcomings, and discuss briefly 
the value of an external evaluation component in that reflective process. 

Self-examination 

Many of the concepts and techniques of self-evaluation derive from traditional 
external evaluation. Useful concepts include maintaining a nonjudgmental orientation, 
soliciting views from the insider's perspective, and triangulating data. The most 
important techniques include observation, interviews, and participation. 

External Evaluation 

Self-examinations and external evaluations, in addition to sharing concepts and 
techniques, can complement each other and help to cross validate data from each 
approach. Self-evaluations help maintain an educational program's health on a daily 
basis; expert external evaluation is essential to an in-depth and objective understanding. 
External evaluators offer training and experience and an "objective eye" rarely found 
inside a program. They can help identify goals and objectives at the onset of a program 
and can help participants take stock of an ongoing program. They can help establish 
standards, benchmarks, and milestones with which to measure student, teacher, 
administrator, and program performance against multiple goals. External evaluators can 
also provide feedback about progress toward those goals and inform policy decision 
makers about the impact of a program in a credible fashion. External evaluation plays an 
invaluable role in refining healthy programs and has a significant impact on future 
funding and programmatic concerns. 

Concluding Thoughts 

Evaluation is essential to learn how a gifted program works, how effective 
programs are, and how to raise their standards of quality. Self-evaluations should be a 
routine part of daily program activity. Students, teachers, administrators, and parents 
should be encouraged to conduct informal self-appraisals on a daily or at least weekly 
basis, questioning and comparing what students are doing in relation to stated program 
goals and objectives. Systems should be developed to give regular feedback to students, 
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teachers, administrators, and parents, including parent-teacher conferences, faculty 
meetings, and student performance conferences. 

External and independent evaluations complement self-evaluations by ensuring a 
more objective and credible appraisal. Formative evaluations provide a continual flow of 
information to program officials throughout a review to improve program practice. 
Summative evaluations can enhance formative evaluations by providing additional 
knowledge with a focus on policy decision making. External evaluations, whether 
qualitative or quantitative, formative or summative can improve program practice and 
student performance. Independent evaluations also help to establish the utility of such 
approaches in gifted education as acceleration, enrichment, and special group settings. 
They are also more credible to sponsors and outside agencies, particularly concerning 
sensitive or controversial issues. 

Nearly all programs can be improved by a critical review. At risk in an 
unexamined program are no less than the health and well-being of gifted and talented 
children and the future of the nation. Together, these approaches play an essential part in 
the development, maintenance, and understanding of educational programs for the gifted 
and talented. 
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EXECUTIVE SUMMARY 

The health of a gifted and talented program requires both self-examination and 
external evaluation. Routine self-examination allows early detection of educational 
problems and confirmation of a sound programmatic approach. By detecting and thus 
helping to prevent program deterioration, self-examination — together with its corollary, 
external evaluation — promotes the intellectual and artistic health of gifted and talented 
education programs. This discussion will highlight some of the common sense ways of 
reflecting upon one's programmatic achievements and shortcomings, and discuss briefly 
the value of an external evaluation component in that reflective process. 

This presentation is designed for multiple audiences. It provides a general 
direction for administrators, encouraging them to conduct self-evaluations and providing 
some guidelines for selecting an outside evaluation expert to assist in the evaluation 
process. It also serves as an orientation for gifted and talented staff members who play a 
role in conducting self-evaluations. Finally, it aims to sensitize trained evaluators new to 
the field of gifted and talented program evaluation to some of the field's nuances. 

This presentation is an overview about evaluation, not a simple how-to instruction 
booklet. Although I am a major proponent of individuals taking evaluation into their own 
hands and conducting self-evaluations, I do not consider one brief discussion sufficient 
preparation for a critical program review. This discussion is a first step, a primer, which 
should be supplemented with reports, texts, classroom instruction, and apprenticeship 
experiences if possible. A variety of recommended texts on evaluation are mentioned 
throughout this presentation. Gifted and talented practitioners new to evaluation should 
seek the assistance of an evaluator to act as coach, assisting in the design and execution 
of an evaluation. 

Self-examination 1 

Many of the concepts and techniques of self-evaluation derive from traditional 
external evaluation. Self-evaluations follow the same general model that external 
evaluators use. A typical model of program assessment involves: 

1. Determining goals and outcomes 

2. Describing the processes required to accomplish goals and 
objectives 

3. Determining the immediate, direct, short-term effects of the 
program (comparing evidence of outcomes) 

4. Determining the long-term effects of the program 
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In addition, evaluation provides useful concepts to guide a self-examination, ^ 
including maintaining a nonjudgmental orientation, soliciting views from the insider's 
perspective, and triangulating data. The most important techniques include observation, 
interviewing, and participation. 

Nonjudgmental Orientation 

A nonjudgmental orientation requires the evaluator to suspend value judgments 
about a given behavior or practice. It is similar to our habit of suspending our disbelief 
while watching a play or a movie: We allow the author a certain latitude and the time to 
unravel the whole story before judging the credibility of the work. A nonjudgmental 
orientation allows the educational evaluator to describe an observed behavior or situation 
in detail— however bizarre or obvious it appears at first glance— before completing an 
analysis of classroom behavior in context. Recommending a nonjudgmental orientation 
in an evaluation may sound contradictory, but such a stance is essential in allowing the 
evaluator to collect a full range of data before screening certain areas out of 
consideration. This orientation is particularly important in an evaluation of gifted 
programs in which novel, creative, and periodically "strange" teaching approaches are 
adopted. For example, a judgmental evaluator watching a classroom of dancing students 
might conclude that the class was involved simply in a dance exercise and thus decide 
that gifted and talented programs are characterized by play rather than content. A 
nonjudgmental evaluator would conduct additional observation and/or interviews that 
could reveal that the class was an eighth-grade physics course engaged in an exciting and 
appropriate approach to demonstrating how long and short light waves operate. A 
judgmental orientation may lead to premature and in many cases inappropriate 
assessments of a specific approach or behavior before there are enough data with which 
to interpret it meaningfully or place it in context. 

Insider's Perspective 

A sound self-examination is grounded in the participants' views of what they are 
doing, what they are trying to accomplish, and what they think they have accomplished. 
Every educational evaluation should attempt to solicit the views of administrators, 
teachers, students, and parents. Even longtime participants in a program can be surprised 
and enlightened by other participants' views on the situation. And when the evaluator' s 
own views are confirmed by interviews with others, the evaluator has strengthened the 
assessment by grounding knowledge in data from others living in and with the situation 
day to day. 

Students, teachers, and administrators may have radically different world views or 
ways of looking at the same educational practices. These different perspectives are a 
natural function of differing roles, values, status, and power. However, when 
perspectives appear irreconcilably different, the parties involved are most likely not 
operating with a system of shared values. Identifying this type of problem, which is often 
at the heart of poorly conceived programs, can explain much of the miscommunication 
observed in a program. Thus, describing a program (including processes and outcomes) 
and the most salient behaviors (including identification practices, teaching, and learning) 
can help in making a determination about whether the practices or behaviors are adaptive 
or maladaptive for students, teachers, administrators, and parents— and why. When 
interviews and observations are grounded in the insiders' views of how a program works, 
findings are considered more salient and recommendations more realistic and practical. 
In addition, the evaluation can identify and contrast multiple program goals with actual 
performance. 




Triangulation 



Triangulation is another invaluable concept tnat any program participant can use 
to guide an effective self-evaluation. Triangulation is used to analyze data. It involves 
testing one source of information against another to strip away alternative explanations. 
For example, an evaluator might think that a gifted and talented program was primarily a 
Humanities-literature oriented program because students were observed discussing 
literature, reciting lines from plays, and critiquing each other's literary reports during 
most of the week. However, a review of the program calendar might indicate that this 
flurry of activity was merely the culmination of the Humanities portion of a much broader 
program. Similarly, a student might say that he is getting straight As. Comparing these 
statements with program records and teacher evaluations not only establishes the 
student's credibility, but more importantly, reveals participants' (often differing) 
perspectives about the program. For example, the evaluator might find corroborating 
evidence that the student is receiving straight As, but might also learn that the grades 
have gone to the student's head, according to a peer. Teachers may indicate that although 
the student is doing very well, in fact no grades are given in the program. In interviews, 
other students may confirm this official policy and then explain how they are informally 
ranked according to ability throughout the program. Thus triangulation may help confirm 
or disprove information and can force the evaluator to probe further to reveal another ^ 
level of information. In both cases, the evaluator learns something about the program s 
effectiveness and about how well it matches its stated philosophy in practice. 

Observation 

There is no adequate substitute for direct, daily observation in evaluation. In a 
self-evaluation, teachers can observe students systematically — for example, observing 
whether different cultural groups interact, whether shy students become more outgoing, 
when a lesson plan is engaging, or whether an administrator participates in teaching or 
social activities associated with the program. Over time, patterns of behavior will 
become evident. These patterns, documented repeatedly over time, are a form of 
reliability. Observations can be used to establish baseline descriptions about the 
program, teacher and student performance, administrative support, and many other 
program-related features. Similarly, observations can be recorded to document change 
(and possibly growth) over time. Observations can be filed in a student portfolio to 
document student performai .ee throughout the year, noting, for example, increasing 
sophistication in research projects, mastery of artistic expression, or complexity of 
mathematical problem solving. Observation is a natural too 1 ; it is enhanced when guided 
by the concepts described above. 

Interviews 

Often the best way to check on or triangulate observation is simply to ask the 
observed participants what they were doing. A student singing in chorus may seem to 
enjoy herself. An interview with that student may reveal, however, that she hates chorus 
and is participating in it only because her parents insisted. Interviews should be open 
ended— that is, they should allow the person being interviewed to answer in is full and 
complete a manner as he or she deems appropriate. Closed yes-or-no questions are quick 
and efficient, but they are usually biased by the person conducting the interview and tend 
to shape the response. Closed questions are useful after initial interviews and 
observations suggest what the relevant questions are — starting from the insider's 
perspective. 
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Participation 



In any educational evaluation, the most credible interviews and observations are 
conducted by evaluators who spend time with program participants. In this sense, 
evaluators who are also participants, who share in the daily lives of students, teachers, 
administrators, and parents, have a much more comprehensive picture of how a gifted and 
talented program operates than the evaluator who simply comes in once in a year, asks a 
few questions, observes a few classes, and then writes a report about the program. There 
is no substitute for the depth of understanding gained from routine participation in 
classroom activity, extracurricular activity, home activity, faculty meetings, and school 
board meetings. Direct participation exposes the evaluator to the multiple levels and 
goals of program participants and affiliates. It also enhances the validity of the 
evaluation findings. 

It is clear from this brief discussion that program participants can — and in many 
cases do — apply evaluation concepts and techniques in their daily lives in gifted and 
talented programs. Observation, measurement, documentation, and evaluation are all 
normal activities of teachers, students, parents, and administrators. Combining these 
concepts and techniques simply helps to make these activities more systematic and 
improves the validity and reliability of the self-evaluation. 

External Evaluation 

Self-examinations and external evaluations, in addition to sharing concepts and 
techniques, can complement each other and help to cross validate data from each 
approach. Self-evaluations help maintain an educational program's health on a daily 
basis; expert external evaluation is essential to an in-depth and objective understanding. 
External evaluators offer training and experience and an "objective eye" rarely found 
inside a program. They can help identify goals and objectives at the onset of a program 
and can help participants take stock of an ongoing program. They can help establish 
standards, benchmarks, and milestones with which to measure student, teacher, 
administrator, and program performance against multiple goals. External evaluators can 
also provide feedback about progress toward those goals and inform policy decision 
makers about the impact of a program in a credible fashion. External evaluation plays an 
invaluable role in refining healthy programs and has a significant impact on future 
funding and programmatic concerns. 

Working With an External Evaluator 

An external evaluator should be chosen as carefully as a personal physician — the 
life of the program may depend on that choice. Choosing a suitable evaluator can also 
help eliminate future problems. For example, a gifted and talented program should be 
developed with the assistance of an evaluator, but it should not be developed to meet the 
evaluator' s needs. Such an approach can skew the program toward easily quantifiable 
elements instead of goals which are more difficult to measure, such as higher order 
thinking goals. Selecting an evaluator who is familiar with gifted and talented programs 
will minimize that kind of problem. Screening and selecting the right evaluator is only 
half the job, however. Participants must also communicate openly with the evaluator 
throughout the evaluation process — even the most talented investigator needs the help of 
participants. 
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In addition, participants need to recognize that the external evaluator must be an 
independent observer. Independence is essential not only to ensure honest and 
reasonably objective feedback about the condition of the program, but to satisfy external 
agencies of the reliability and validity of the evaluator' s finding. Sponsors typically 
demand an external evaluation to determine whether taxpayer money is being spent 
prudently and whether the program is educationally on track. 

Finally, participants need to listen carefully to the external evaluator' s assessment. 
This does not mean that a program should blindly follow any and all recommendations. 
However, ignoring sound — even if unpleasant — advice can have serious consequences. 
Unchecked, administrative and curricular problems can become deeply ingrained in the 
body of the program, compounding the difficulties year after year. External evaluations 
are more expensive than self-evaluations, but like an annual physical, they are worth the 
time and investment. 

Qualitative and Quantitative Approaches 

External evaluations, like self-evaluations, should combine qualitative and 
quantitative approaches. Qualitative approaches are used to describe a program, its 
operation, and its effects. They are also required to establish the appropriate baseline or 
context within which to interpret student performance. An accurate measure of student 
performance requires preentry information about student, program, and community 
characteristics, including cultural, social, political, and economic factors. For example, it 
is important to document whether the evaluation focuses on a program that is just starting 
up or is a mature program operating for years. In addition, an accurate measure of how 
far a student has traveled requires qualitative and quantitative baseline data about the 
student's previous attendance patterns, grades, reading level, achievement record, and so 
on. Such data can also indicate whether a student is underachieving. In addition, 
qualitative approaches (such as ethnography, an established qualitative approach in 
evaluation) describe program implementation in sufficient detail to explicate the multiple 
goals of a program; delineate the process in which students, teachers, administrators, and 
parents interact (in school and at home); and highlight adaptive and maladaptive social 
arrangements. 

A description of the program can determine whether or not student identification 
techniques and program approaches match. In addition, a description of the process of 
how students and teachers function enables program and policy officials to mfake program 
modifications that can enhance student outcomes. Gifted and talented programs ane 
particularly amenable to the use of qualitative approaches, in that creative approaches to 
teaching can be described and documented with minimal obtrusiveness. 

Qualitative information can also be used to document program effects. The data 
are readily available in gifted programs and can contribute to a more objective and useful 
evaluation. For example, in an arts-oriented gifted program a portfolio of the student's 
accomplishments or products can be rated from beginner to professional quality. This 
would produce useful entry, mid-course, and end-of-year program data. In addition to 
documenting outcomes over time, the same data can be compared to program description 
data to draw causal inferences about specific elements of the program, such as teacher 
dedication, and specific outcomes, such as student participation in mathematical 
competitions. Similarly, teams can be assembled to rate academic portfolios throughout 
the year — focusing on pre-established criteria, including mastery in linguistic and 
scientific domains. 
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Converting qualitative data about program variables into a quantitative format can 
facilitate comparison and analysis of program effects. For example, student participation 
in the program could be translated or converted into zero to five days a week, concerning 
gender — males could be coded as one and females as two in a binary fashion, time of 
day — morning coded one and afternoon two, topic — English coded one and mathematics 
two, and specific instructional approach — lecture coded one and team teaching two. 
This simple qualitative/quantitative conversion would facilitate analysis of the qualitative 
data, enabling the evaluator to determine the effects of a program including differential 
effects on different students. This example alone would provide an insight into general 
program outcomes, as well as effects of different amounts or types of instruction, student 
gender, and time of day. It would also describe how these variables interacted. Using 
qualitative data to describe program effects is one of the most promising (and needed) 
approaches to developing more meaningful evaluations of gifted and talented programs. 

Quantitative measures document outcomes, including student achievement over 
time. Standardized tests have been considered the "coin of the realm" for many years in 
evaluation, and they are relied on extensively in gifted program evaluation. However, the 
methodological tail should not wag the dog. The most appropriate approach for the task 
at hand should be selected. Comparisons of pre- and post-test standardized achievement 
scores of gifted and talented children do not produce significant outcomes and should not 
be expected to produce any if the students are being selected appropriately. 2 Moreover, 
outcomes alone are worthless without some indication of whether students are achieving 
stated goals and objectives. There are many forms of quantitative data, other than 
standardized tests, that can document the effectiveness of gifted programs, including the 
number of and frequency of participation in intellectually competitive activities — such as 
Science Fairs, Westinghouse, Putnam (math), and other engaging enterprises— as well as 
more mundane figures such as attendance, books read, portraits and recitals completed, 
number entering post-secondary institutions, and so on. Per-pupil cost figures contribute 
to a cost-effectiveness analysis. Longitudinal follow-up data on student academic and 
employment careers provide some insight into program impact. Together, qualitative and 
quantitative approaches effectively document and analyze evaluation data. (See 
Coleman. 1985; Fetterman, 1988a; Gallagher 1985; Howley, Howley, & Pendarvis, 1986; 
Kitano & Kirby, 1986; Reis & Renzulli, 1991; Renzulli, 1975; Renzulli & Ward, 1969 
for a detailed discussion about evaluating gifted and talented education programs. In 
addition, see Fetterman, 1988b for a discussion about the variety of qualitative 
approaches to evaluation in education.) 

Communication and Summing Up 

Writing is part of the analysis process; it clarifies and organizes thought and helps 
to specify ideas and relationships (see Fetterman, 1989; Hammersley & Atkinson, 1983). 
In some instances evaluators discover in the process of writing that they have gaps in 
their knowledge. If they are still in the process of (or involved in) reviewing a gifted and 
talented education program, they have the opportunity to conduct additional interviews 
and observe specific settings. If they have left the school setting, they have only their 
records and notes to rely on. Frequently embryonic ideas come to maturity during 
writing, as the evaluator is forced to crystallize his or her thinking on a particular topic. 
The reporting of findings usually takes place in memoranda and a final report. An 
interim report may prove a useful tool as well. 



xvi?5 



Communication and Program Practice 



Evaluators must communicate their findings and recommendations if they are 
going to be effective. Verbal communications and memoranda issued throughout an 
evaluation keep evaluators and program personnel on track and provide a quality control 
on the data while the evaluation is still in progress. (By the time the final report is 
written, it is often too late to correct erroneous ideas that have been used to build the 
conceptual foundation of the evaluation findings.) Interim reports are valuable forms of 
communication for program participants, sponsors, and evaluators. They create 
benchmarks in the study and provide interested parties with findings in progress, current 
thinking, and a progress report on the evolving direction of the evaluation. All of these 
interim forms of communication minimize surprises, gross errors, and related problems. 
The final report puts a cap on the entire experience for the evaluator, participants, and 
sponsor and typically includes recommendations that, if used properly, will serve as a 
guide to improve program practice. 

Concluding Thoughts 

Evaluation is needed to learn how a gifted program works, how effective 
programs are, and how to raise their standards of quality. Self-evaluations should be a 
routine part of daily program activity. Students, teachers, administrators, and parents 
should be encouraged to conduct informal self -appraisals on a daily or at least weekly 
basis, questioning and comparing what students are doing in relation to stated program 
goals and objectives. Systems should be developed to give regular feedback to students, 
teachers, administrators, and parents, including parent-teacher conferences, faculty 
meetings, and student performance conferences. 

In addition, external and independent evaluations ensure a more objective and 
credible appraisal. Formative 3 evaluations provide a continual flow of information to 
program officials throughout a review to improve program practice. Summative 4 
evaluations can complement formative evaluations. The knowledge gained at the end of 
a summative evaluation culminates in a final report with a focus on policy decision 
making. Evaluations, whether qualitative or quantitative, formative or summative can 
improve program practice and student performance. Independent evaluations are also 
required to establish the utility of such approaches in gifted education as acceleration, 
enrichment, and special group settings. 

Few programs can't be improved by a critical review. If unexamined, the health 
and well-being of gifted and talented children and the future of the nation are at risk. 
Together, these approaches play an essential part in the development, maintenance, and 
understanding of educational programs for the gifted and talented. (A more detailed 
discussion about these concepts, techniques, and approaches is provided in the full report 
which can be obtained by contacting The National Research Center on the Gifted and 
Talented.) 

Case Study: An Illustration 

An annotated case study of an evaluation of a gifted and talented program 
appeared in Excellence and Equality: A Qualitatively Different Perspective on Gifted and 
Talented Education (Fettenran, 1988). This case study highlighted many of the methods, 
concepts, and techniques presented in this discussion. In addition, it demonstrated the 
scope of evaluation. This evaluation included an analysis of the referral, identification, 
and selection mechanisms, as well as a review of the entire program operation. It also 
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served as a national test case for gifted and talented education and minority enrollment. 
Fundamentally, the evaluation addressed the issues of equal opportunity, ability, and 
achievement in American education. 
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General Evaluation Guidelines 



Guideline One: Make sure the evaluation serves the practical information needed 
by the targeted audiences. 

Discussion: Take time to identify who the key stakeholders are including administrators, 
parents, teachers, and students. Find out what they want to know and then develop a plan 
to address the most salient concerns. 

Guideline Two: Make sure the evaluation is realistic (politically and pragmatically) 
and cost effective. 

Discussion: There is no point in conducting an evaluation if there is no political support 
for a program or if there is good reason to believe the findings will be hidden or altered. 
Care must also be taken to ensure that an evaluation effort does not significantly drain 
program resources. 

Guideline Three: Make sure that the evaluation is conducted in an ethical manner. 

Discussion: The rights of program participants must be protected. Care should be taken 
to ensure privacy, freedom of information, and confidentiality (if promised). 

Guideline Four: Make sure the evaluation is as accurate as possible. 

Discussion: Take the time to seek out various sources of information and cross-check 
data sources. Double check all figures and interview notes. Judgments about the data 
should be logical and reviewed by independent sources whenever possible. 

For a more detailed discussion of evaluation standards and guidelines see The Program 
Evaluation Standards by the Joint Committee on Standards for Educational Evaluation 
(1992) and Rossi's Standards for Evaluation Practice (1982). 

Specific Guidelines for Evaluating Gifted and Talented 
Education Programs 

Guideline One: Make sure program documentation exists. 

Discussion: Program documentation should describe the program's philosophy; 
curriculum; staffing; financial, library, and computer resources; identification and 
screening procedures; and selection criteria. In addition, classroom schedules and maps 
of the physical layout will facilitate any evaluation. 

Guideline Two: Make sure you review as many relevant data sources as possible. 

Discussion: Interviews and observations are critical. In addition, archival 
documentation, such as newsletters, financial reports, student letters, parent letters, past 
evaluation reports, newspaper articles, and many other documents provide pertinent data 
about the program's impact and role in the community. 
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Guideline Three: Make sure you compare the program's stated goals with their 
actual performance* 

D iscussion: Does the program operate in accordance with its own philosophy 
(academically and in terms of governance)? Does the curriculum reflect the philosophy 
and goals of the school? Do the staff members appear to understand and implement the 
stated program philosophy? How do teachers translate the program's philosophy into 
practice in their teaching? 

Guideline Four: Make sure you describe and assess the climate. 

Discussion: Are students engaged? Are teachers stimulating, thoughtful, and 
knowledgeable? Is communication between staff and administration constructive and 
cooperative or antagonistic and fragmented? Similarly, what is the nature and tone of 
communication with and among students. 

Guideline Five: Make sure you talk to students. 

Discussion: The purpose of gifted and talented education programs is to serve students. 
Time should be devoted to informally interview students about their own progress 
(including a review of their portfolio, records, or projects) and their evaluation of the 
program. Student academic achievement and behavior code data are critical to any gifted 
and talented education program evaluation. 

Guideline Six: Make sure program finances are reviewed. 

Discussion: Is the program budget sufficient, if not, why not? Is the money being used 
as intended, if not, why not? Is financial program planning adequate and appropriate to 
meet the needs of the program in the foreseeable future? 

Guideline Seven: Make sure community and school board components are included 
in the evaluation. 

Discussion: Do community and school board members support the program? What is 
the evidence? Do parents participate in the program? What are the obstacles to 
community and board support if any? 
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Notes 



1 During my presidency of the American Evaluation Association, I created the 
concept empowerment evaluation. This involves using evaluation to help others help 
themselves. One facet of this approach involves encouraging program participants to 
conduct their own evaluations and to share useful models and concepts with them to help 
them help themselves. 

2 However, they can be used to document change over time in student 
performance, particularly for underachievers and students from culturally diverse 
backgrounds. 

3 Scriven refers to formative evaluation as evaluation aimed at the improvement 
of an ongoing enterprise (see Davis, Scriven, & Thomas, 1981). 

4 According to Davis, Scriven, and Thomas (1981), summative evaluation is 
conducted for an external client and its primary purpose is to report on the quality of the 
program for purposes other than improvement 
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Introduction 



Gifted and talented education programs, 1 perhaps more acutely than most 
educational programs, require a clearsighted and self-critical awareness of program 
strengths and goals to promote intellectual and artistic growth. Such awareness is 
fostered through an ongoing process of self-evaluation. Administrators, teachers, 
counselors, parents, and students, working as a team, can play an active role in 
evaluating, shaping, and continually improving their program by taking a critical look at 
program achievements, objectives, and the day-to-day realities of program performance. 
This discussion will highlight some of the common sense ways of reflecting upon one's 
programmatic achievements and shortcomings, and discuss briefly the value of an 
external evaluation component in that reflective process. It will also provide standard 
questions to ask in the process of an evaluation and outline both general evaluation 
guidelines and guidelines specific to the evaluation of gifted and talented education 
programs. These guidelines can be adapted to the individual local context. A detailed 
case example is presented to illustrate the scope and multifaceted nature of a program 
evaluation. In addition, it highlights some of the methods and concepts discussed in this 
presentation. 

This presentation is designed for multiple audiences. It provides a general 
direction for administrators, encouraging them to conduct self-evaluations and providing 
some guidelines for selecting an outside evaluation expert to assist in the evaluation 
process. It also serves as an orientation for gifted and talented staff members who play a 
role in conducting self-evaluations. Finally, it aims to sensitize trained evaluators new to 
the field of gifted and talented program evaluation to some of the field's nuances. 

This presentation is an overview about evaluation, not a simple how-to instruction 
booklet. Although I am a major proponent of individuals taking evaluation into their own 
hands and conducting self-evaluations, I do not consider one brief discussion sufficient 
preparation for a critical program review. This discussion is a first step, a primer, which 
should be supplemented with reports, texts, classroom instruction, and apprenticeship 
experiences if possible. A variety of recommended texts on evaluation are mentioned 
throughout this presentation. Gifted and talented practitioners new to evaluation should 
seek the assistance of an evaluator to act as coach, assisting in the design and execution 
of an evaluation. 

A word or two about the staffing structure or configuration of evaluations is in 
order before launching into critical evaluation concepts and techniques. The team 
approach is the most common and often the most effective method of conducting self- or 
external evaluations. A typical self-evaluation team is composed of teachers, counselors, 
administrators, parents, and a student. Local experts, board members, and gifted and 
talented program officials are recruited to serve whenever possible. The team develops 
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an evaluation plan, based on input from key stakeholders in the school and community. 
Evaluation tasks — including interviews, observations, and reviews of records — are 
divided and distributed to appropriate team members, who execute them. The team 
assembles at various points in the process to share notes, report on progress, and suggest 
areas requiring additional investigation. Team members explore new leads and cross- 
check each other's data. The team chair is typically charged with the responsibility of 
producing an evaluation report. The team approach makes it possible to conduct a full- 
scale evaluation within the tight time and budget constraints typical of an educational 
program. It also ensures a balanced perspective, as team members cross-validate each 
others' judgments and observations, providing a continual check against individual 
observer bias. 

An external evaluation team has a similar structure and process. However, an 
external evaluation team is typically composed of individuals outside the program, 
including other gifted and talented program directors, practitioner-scholars in the field, 
teachers, and community members. Parents and students may also be recruited to join the 
evaluation team. 

Self-examination 2 



Many of the concepts and techniques of self-evaluation derive from traditional 
external evaluation. Self-evaluations follow the same general model that external 
evaluations use. In addition, evaluation provides useful concepts to guide an effective 
self-examination, including maintaining a nonjudgmental orientation, soliciting views 
from the insider's perspective, and triangulating data. 3 Evaluation concepts underlie the 
most important techniques, including observation, interviewing, and participation. These 
concepts provide the lens through which observations are made and interview questions 
are asked in the field. Evaluation concepts and techniques are useful guides in any 
program inquiry, from a study of pragmatic concerns about fiscal resources to an 
investigation of larger social issues such as minority enrollment. 

Evaluation Model 

A typical model of program self-assessment, derived from traditional evaluations, 
involves: 

1. Determining goals and outcomes 

2. Describing the processes required to accomplish goals and 
objectives 

3. Determining the immediate, direct, short-term effects of the 
program (comparing evidence of outcomes) 

4. Determining the long-term effects of the program 

A conscientious effort should be made to determine the goals of the specific gifted 
program. The goals should be specific, realistic, and measurable. For example, an 
artistic program may establish a goal of every student performing a recital by the end of 
the year. An academic math/science program might select as goals having a certain 
percentage of the student body enter and win the Westinghouse Science Talent Search 
and Putnam Mathematical competitions. Goals can range from having students perform 
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at two grades levels beyond their chronological peers, to having them volunteer 400 
hours to public service projects or conduct research, literary, or dramatic projects. 
Whatever the goal, it is important for the evaluators to recognize the program s explicit 
target outcomes before undertaking an evaluation of that program. 

The second major step is to describe what the program is doing to accomplish its 
stated objectives. This requires continual monitoring of program operations, as well as 
discussions with administrators, teachers, and students. Observations of this nature will 
help shape questions about accountability and help explain why the program is or isn t 
working In addition, monitoring the program on a routine basis helps administrators, 
teachers, and students keep it on course— aimed at accomplishing the stated objectives of 
the program. Ideally, evaluation data can be used to improve program operations and 
eliminate ineffective and wasteful facets of the program. 

The third step is to determine the program's immediate or direct impact. Are 
gifted and talented students performing at two grades levels beyond their chronological 
peers'* Are students entering science and mathematical competitions at the numbers and 
level desired or expected? Are teachers providing a qualitatively different educational 
experience for gifted and talented students, in terms of depth and pace of study? Useful 
sources of information for this stage of the evaluation include interviews with 
administrators, teachers, students, and parents, observations of administrative and 
classroom behavior, and archival data such as past evaluation reports, standardized test 
results student and program portfolios, and local newspaper articles. This is the stage of 
an evaluation that requires particular attention to validity 4 and reliability. These concepts 
are described in depth in standard evaluation and measurement textbooks. In general, the 
evaluator must be sure to select an accurate and valid measure to determine whether a 
goal has been achieved. Such a measure can be scores on achievement tests, community 
service records, or a dramatic production. The measure selected should also be reliable or 
stable— in other words, it should provide the same information about a specific situation 
or set of circumstances over time (assuming nothing has changed about the situation). 

A final step is to consider the program's long-term or ultimate effect. Has the 
gifted and talented program contributed to the academic standing of the school in the 
community, the state, and the nation? Are more gifted and talented students entering and 
completing undergraduate and graduate degree programs? Are more gifted and talented 
program graduates making productive business, medical, or scientific contributions ! 
Typically few programs maintain comprehensive or systematic longitudinal data on their 
graduates. However, such data provide one of the best and least expensive sources of 
useful impact information. 

Evaluation Concepts and Techniques 
Nonjudgmental Orientation 

A noniudgmental orientation requires the evaluator to suspend value judgments 
about a given behavior or practice. It is similar to our habit of suspending our disbelief 
while watching a play or a movie: We allow the author a certain latituue and the time to 
unravel the whole story before judging the credibility of the work. A nonjudgmental 
orientation allows the educational evaluator to describe an observed behavior or situation 
in detail— however bizarre or obvious it appears at first glance— before completing an 
analysis of classroom behavior in context. Recommending a nonjudgmental orientation 
in an evaluation may sound contradictory, but such a stance is essential in allowing the 
evaluator to collect a full range of data before screening certain areas out of 
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consideration. This orientation is particularly important in an evaluation of gifted 
programs, in which novel, creative, and periodically "strange" teaching approaches are 
adopted. For example, a judgmental evaluator watching a classroom of dancing students 
might assume that the class was involved simply in a dance exercise and thus conclude 
that gifted and talented programs are characterized by play rather than content. A 
nonjudgmental evaluator would conduct additional observation and/or interviews that 
could reveal that the class was an eighth-grade physics course engaged in an exciting and 
appropriate approach to demonstrating how long and short light waves operate. A 
nonjudgmental orientation thus prevents premature and in many cases inappropriate 
assessments of a specific approach or behavior and ensures that there are enough data to 
interpret it meaningfully or place it in context. (See Appendix A for additional discussion 
about a nonjudgmental orientation.) 

Insider's Perspective 

A sound self-examination is grounded in the participants' views of what they are 
doing, what they are trying to accomplish, and what they think they have accomplished. 
Every educational evaluation should attempt to solicit the views of administrators, 
teachers, students, and parents. Even longtime participants in a program can be surprised 
and enlightened by other participants' views on the situation. And when the evaluator's 
own views are confirmed by interviews with others, the evaluator has strengthened the 
assessment by grounding knowledge in data from others living in and with the situation 
day to day. 

Students, teachers, and administrators may have radically different world views or 
ways of looking at the same educational practices. These different perspectives are a 
natural function of differing roles, values, status, and power. However, when 
perspectives appear irreconcilably different, the parties involved are most likely not 
operating with a system of shared values. Identifying this type of problem, which is often 
£t the heart of poorly conceived programs, can explain much of the miscommunication 
observed in a program. Thus, describing a program (including processes and outcomes) 
and the most salient behaviors (including identification practices, teaching, and learning) 
can help in making a determination about whether the practices or behaviors are adaptive 
or maladaptive for students, teachers, administrators, and parents — and why. When 
interviews and observations are grounded in the insiders' views of how a program works, 
findings are considered more salient and recommendations more realistic and practical. 
In addition, the evaluation can identify and contrast multiple program goals with actual 
performance. 5 (See Appendix B for additional discussion about the insider's 
perspective.) 

Triangulation 

Triangulation is another invaluable concept that any program participant can use 
to guide an effective self-evaluation. Triangulation is used to analyze data. It involves 
testing one source of information against another to strip away alternative explanations. 
For example, an evaluator might think that a gifted and talented program was primarily a 
Humanities-literature oriented program because students were observed discussing 
literature, reciting lines from plays, and critiquing each other's literary reports during 
most of the week. However, a review of the program calendar might indicate that this 
flurry of activity was merely the culmination of the Humanities portion of a much broader 
program. Similarly, a student might say that he is getting straight As. Comparing these 
statements with program records and teacher evaluations not only establishes the 
student's credibility, but more importantly, reveals participants' (often differing) 



27 



perspectives about the program. For example, the evaluator might find corroborating 
evidence that the student is receiving straight As, but might also leam that the grades 
have gone to the student's head, according to a peer. Teachers may indicate that although 
the student is doing very well, in fact no grades are given in the program. In interviews, 
other students may confirm this official policy and then explain how they are informally 
ranked according to ability throughout the program. Thus triangulation may help confirm 
or disprove information and can force the evaluator to probe further to reveal another 
level of information. 6 In both cases, the evaluator learns something about the program's 
effectiveness and about how well it matches its stated philosophy in practice. 7 

Triangulation is at the cornerstone of every gifted and talented education program 
evaluation. This approach requires the collection of basic program documentation, such 
as statements describing the program's philosophy, curriculum, staffing, finances, as well 
as information about identification, screening, and selection criteria. Then the evaluator 
must collect additional data from observations and interviews and other sources to 
compare it with the stated policies, goals, and objectives. Triangulation allows the 
evaluator to penetrate the veneer of a program. It also allows program participants to see 
themselves as they really are as compared with how they would like to see themselves. 
In this process, evaluation can help program participants approximate their goals and 
objectives and thus more closely resemble their idealized picture of themselves. (See 
Appendix C for additional discussion about triangulation.) 

Observation 

There is no adequate substitute for direct, daily observation in evaluation. In a 
self-evaluation, teachers can observe students systematically— for example, observing 
whether different cultural groups interact, whether shy students become more outgoing, 
when a lesson plan is engaging, or whether an administrator participates in teaching or 
social activities associated with the program. Over time, patterns of behavior will 
become evident. These patterns, documented repeatedly over time, are a form of 
reliability. Observations can be used to establish baseline descriptions about the 
program, teacher and student performance, administrative support, and many other 
program-related features. Similarly, observations can be recorded to document change 
(and possibly growth) over time. Observations can be filed in a student portfolio to 
document student performance throughout the year, noting, for example, increasing 
sophistication in research projects, mastery of artistic expression, or complexity of 
mathematical problem solving. 

Portfolio reviews should determine if any documentation is being maintained, if it 
is being maintained on a regular basis, and if it is relevant or appropriate. Progressive 
stages of development should be discernible in a portfolio. For example, summaries of 
student research projects with increasing levels of complexity and sophistication should 
be maintained and dated in the file, documentation of mastery over specific art forms 
should be compiled in a student portfolio, verbatim quotations about student self- 
assessment and teacher assessments of students should be maintained on a regular basis, 
newspaper articles about specific student accomplishments should be readily available, 
and records (and results) of participation in contests and award ceremonies should be 
compiled. These documents provide ?ji ongoing record of student performance and an 
excellent data bank for evaluators. On-site observation is useful to help compile student 
portfolio data and to verify that the information is useful and accurate. 

Observations are useful in the discovery, compilation, and verification of data. 
The simple act of sitting in the classroom observing instructional activities may provide 
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such insights as: real attendance as compared with attendance figures, minority 
enrollment, defacto segregation, student engagement, instructional philosophy and 
practice, use and availability of school and community resources, and many other 
significant indicators of program practice. Observations can provide data about student 
behavior code patterns, quality of the physical plant (including delayed maintenance) and 
parental participation. Observations of communication patterns in the classrooms, the 
hallways, the teachers' lounge, or the playground may also be indicative of either 
adaptive or maladaptive instructional styles or governance practices. Triangulated data 
from observations, interviews, and physical indicators (like the presence or absence of 
awards or graffiti) can provide a useful insight into the school's climate. Observation is a 
natural tool; it is enhanced when guided by the concepts described above. (See Appendix 
D for additional discussion about participant observation.) 

Interviews 

Often the best way to check on or triangulate observation is simply to ask the 
observed participants what they were doing. A student singing in chorus may seem to 
enjoy herself. An interview with that student may reveal, however, that she hates chorus 
and is participating in it only because her parents insist Interviews should be open 
ended — that is, they should allow the person being interviewed to answer in as full and 
complete a manner as he or she deems appropriate. Closed yes-or-no questions are quick 
and efficient, but they are usually biased by the person conducting the interview and tend 
to shape the response. Closed questions are useful after initial interviews and 
observations suggest what the relevant questions are — starting from the insider's 
perspective. (See Appendix E for additional discussion about interviews.) 

Evaluation team members should interview key stakeholders and other team 
members to construct a useful list of questions to evaluate a gifted and talented education 
program. A list of standard questions to guide a gifted and talented education program 
evaluation has been provided below. These questions have evolved from more open- 
ended questions. This list is neither exhaustive nor all-inclusive; it is simply designed to 
guide an evaluation effort. These generic questions should be tailored to the specific 
audiences that are most appropriate, using the concepts and techniques described above. 

Evaluation Questions 

1. Is there any program documentation about the philosophy, curriculum, 
finances, and staffing about the program? 

2. Is there any documentation about the identification, screening, and 
selection criteria? 

3. Are the identification, screening, and selection criteria appropriate fur the 
program in operation? (Typical programs have an academic focus — for 
example, humanities, mathematics, or science; some programs focus on 
visual and performing arts.) 

4. Has the program conducted either internal or external evaluations? Have 
these records been maintained? What were the findings and 
recommendations? Were the recommendations followed, and why or why 
not? 

5. Does the program operate in accordance with its own philosophy? 
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6. Does the curriculum reflect the philosophy and goals of the school 
program? 

7. Do staff members understand and implement the stated program 
philosophy? 

8. Do staff members work well together? 

9. Do they think they are provided with adequate and appropriate support, 
ranging from supplies to compensation packages? What preparation time 
and professional development funds are available for teachers? Do 
teachers find them adequate or appropriate? 

10. Are students engaged? Is there any observation, product, interview, or 
other documentation of critical and creative thinking in the program? 

1 1. What do students think about the program? Do they like one topic or 
teacher more than another? Why? 

12. What does the budget look like? How was it developed? Is it adequate? 

13. What are the level and quality of community support or interaction? Are 
there obstacles to community or board support? 

Participation 

In any educational evaluation, the most credible interviews and observations are 
conducted by evaluators who spend time with program participants. In this sense, 
evaluators who are also participants, who share in the daily lives of students, teachers, 
administrators, and parents, have a much more comprehensive picture of how a gifted and 
talented program operates than the evaluator who simply comes in once in a year, asks a 
few questions, observes a few classes, and then writes a report about the program. There 
is no substitute for the depth of understanding gained from routine participation in 
classroom activity, extracurricular activity, home activity, faculty meetings, and school 
board meetings. Direct participation exposes the evaluator to the multiple levels and 
goals of program participants and affiliates. It also enhances the validity of the 
evaluation findings. 8 

It is clear from this brief discussion that program participants can — and in many 
cases do—apply evaluation concepts and techniques in their daily lives in gifted and 
talented programs. Observation, measurement, documentation, and evaluation are all 
normal activities of teachers, students, parents, and administrators. Combining these 
concepts and techniques simply helps to make these activities more systematic znd 
improves the validity and reliability of the self-evaluation. 

External Evaluation 



Self-examinations and external evaluations, in addition to sharing concepts and 
techniques, can complement each other and help to cross validate data from each 
approach. Self-evaluations help maintain an educational program's health on a daily 
basis; expert external evaluation is essential to an in-depth and objective understanding. 
External evaluators offer training and experience and an "objective eye" rarely found 
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inside a program. They can help identify goals and objectives at the onset of a program 
and can help participants take stock of an ongoing program. They can help establish 
standards, benchmarks, and milestones with which to measure student, teacher, 
administrator, and program performance against multiple goals. External evaluators can 
also provide feedback about progress toward those goals and inform policy decision 
makers about the impact of a program in a credible fashion. External evaluation plays an 
invaluable role in refining healthy programs and has a significant impact on future 
funding and programmatic concerns. 

Working With an External Evaluator 

An external evaluator should be chosen as carefully as a personal physician — the 
life of the program may depend on that choice. Choosing a suitable evaluator can also 
help eliminate future problems. For example, a gifted and talented program should be 
developed with the assistance of an evaluator, but it should not be developed to meet the 
evaluator' s needs. Such an approach can skew the program toward easily quantifiable 
elements instead of goals which are more difficult to measure, such as higher order 
thinking goals. Selecting an evaluator who is familiar with gifted and talented programs 
will minimize that kind of problem. Screening and selecting the right evaluator is only 
half the job, however. Participants must also communicate openly with the evaluator 
throughout the evaluation process — even the most talented investigator needs the help of 
participants. 

In addition, participants need to recognize that the external evaluator must be an 
independent observer. Independence is essential not only to ensure honest and 
reasonably objective feedback about the condition of the program, but to satisfy external 
agencies of the reliability and validity of the evaluator' s finding. Sponsors typically 
demand an external evaluation to determine whether taxpayer money is being spent 
prudently and whether the program is educationally on track. 

Finally, participants need to listen carefully to the external evaluator' s assessment. 
This does not mean that a program should blindly follow any and all recommendations. 
However, ignoring sound — even if unpleasant — advice can have serious consequences. 
Unchecked, administrative and curricular problems can become deeply ingrained in the 
body of the program, compounding the difficulties year after year. External evaluations 
are more expensive than self-evaluations, but like an annual physical, they are worth the 
time and investment. 

Qualitative and Quantitative Approaches 

External evaluations, like self-evaluations, should combine qualitative and 
quantitative approaches. Qualitative approaches are used to describe a program, its 
operation, and its effects. They are also required to establish the appropriate baseline or 
context within which to interpret student performance. An accurate measure of student 
performance requires preentry information about student, program, and community 
characteristics, including cultural, social, political, and economic factors. For example, it 
is important to document whether the evaluation focuses on a program that is just starting 
up or is a mature program operating for years. In addition, an accurate measure of how 
far a student has traveled requires qualitative and quantitative baseline data about the 
student's previous attendance patterns, grades, reading level, achievement record, and so 
on. Such data can also indicate whether a student is underachieving. 9 In addition, 
qualitative approaches (such as ethnography, an established qualitative approach in 
evaluation) describe program implementation in sufficient detail to explicate the multiple 
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goals of a program; delineate the process in which students, teachers, administrators, and 
parents interact (in school and at home); and highlight adaptive and maladaptive social 
arr angements. (See Fetterman, 1989 for a step-by-step guide to this process.) 

A description of the program can determine whether or not student identification 
techniques and program approaches match. 10 In addition, a description of the process of 
how students and teachers function enables program and policy officials to make program 
modifications that can enhance student outcomes. Gifted and talented programs are 
particularly amenable to the use of qualitative approaches, in that creative approaches to 
teaching can be described and documented with minimal obtrusiveness. 

Qualitative information can also be used to document program effects. The data 
are readily available in gifted programs and can contribute to a more objective and useful 
evaluation. For example, in an arts-oriented gifted program a portfolio of the student's 
accomplishments or products can be rated from beginner to professional quality. This 
would produce useful entry, mid-course, and end-of-year program data. In addition to 
documenting outcomes over time, the same data can be compared to program description 
data to draw causal inferences about specific elements of the program, such as teacher 
dedication, and specific outcomes, such as student participation in mathematical 
competitions. 1 1 Similarly, teams can be assembled to rate academic portfolios 
throughout the year — focusing on pre-established criteria, including mastery in linguistic 
and scientific domains. 

Converting qualitative data about program variables into a quantitative format can 
facilitate comparison and analysis of program effects. For example, student participation 
in the program could be translated or converted into zero to five days a week, concerning 
gender — males could be coded as one and females as two in a binary fashion, time of 
day — morning coded one and afternoon two, topic — English coded one and mathematics 
two, and specific instructional approach — lecture coded one and team teaching two. 
This simple qualitative/quantitative conversion would facilitate analysis of the qualitative 
data, enabling the evaluator to determine the effects of a program including differential 
effects on different students. This example alone would provide an insight into general 
program outcomes, as well as effects of different amounts or types of instruction, student 
gender, and time of day. It would also describe how these variables interacted. Using 
qualitative data to describe program effects is one of the most promising (and needed) 
approaches to developing more meaningful evaluations of gifted and talented programs. 12 
(See Miles & Huberman, 1984 for a detailed discussion about qualitative data analysis.) 

Quantitative measures document outcomes, including student achievement over 
time. Standardized tests have been considered the "coin of the realm" for many years in 
evaluation, and they are relied on extensively in gifted program evaluation. However, the 
methodological tail should not wag the dog. The most appropriate approach for the task 
at hand should be selected. Comparisons of pre- and post-test standardized achievement 
scores of gifted and talented children in an academically-oriented program do not 
produce significant outcomes and should not be expected to produce any if the students 
are being selected appropriately. 13 Moreover, outcomes alone are worthless without 
some indication of whether students are achieving stated goals and objectives. There are 
many forms of quantitative data, other than standardized tests, that can document the 
effectiveness of gifted programs, including the number and frequency of participation in 
intellectually competitive activities— such as science fairs, the Westinghouse Science 
competition, Putnam (math) contest, and other engaging enterprises — as well as more 
mundane figures such as attendance, books read, portraits and recitals completed, number 
entering post-secondary institutions, and so on. Contract systems provide useful 
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quantifiable outcome data that is easily verifiable, concerning the extent to which 
students are achieving stated goals and objectives. Teacher and student survey 
questionnaires and rating scales can be developed that focus on specific topics. 

For example, an open alternative classroom survey might ask the following 



questions: 




1. 


Does the teacher provide multiple resources? 


2. 


Is the curriculum driven by a collaborative effort? 


3. 


Does the teacher provide varied, interactive learning experiences? 


4. 


Does the teacher provide cooperative learning experiences? 


5. 


Does the teacher facilitate student self-evaluation? 


6. 


Does the teacher identify student learning styles? 


7. 


Does the teacher communicate with multiple groups? 


8. 


Does the teacher encourage student decision-making? 


9. 


Does the teacher expect critical-creative thinking? 



Student ratings of teachers can focus on whether assignments are clear, instruction is 
challenging, the teacher listens to the students, the teacher is enthusiastic about the 
subject, the teacher is knowledgeable about the subject, and so on. A survey about 
students might ask: 

1. Is the student involved or engaged in classroom activity without 
continuous supervision? 

2. Does the student have curricular options? 

3. Does the student continue to study in the middle of other student activity? 

4. Does the student interact constructively with other students? 

5. Do students assess themselves and maintain records of their progress? 

6. Does the student have the appropriate materials and support services to 
function effectively? 

7. Does the student communicate effectively with the teacher? 

8. Does the student work with other students? 

9. Does the student exhibit critical-creative thinking? 

All these questions, including the classroom survey, student survey, and student ratings of 
teachers, can be rated according to a five-point scale and the distributions analyzed. 
(Minimally, yes and no responses would be supplemented with explanations of why or 
why not as appropriate.) Patterns can be identified to reveal perceptions about the 
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program's overall effectiveness and specific strengths and weaknesses. Per-pupil cost 
figures contribute to a cost-benefit analysis. (See Levin, 1983, for a detailed discussion 
about conducting a cost-benefit and cost-effectiveness analysis in an educational setting.) 
Longitudinal follow-up data on student academic and employment careers provide insight 
into program impact. Together, qualitative and quantitative approaches effectively 
document and analyze evaluation data. (See Coleman, 1985; Fetterman, 1988a; 
Gallagher, 1985; Howley et al, 1986; Kitano & Kirby, 1986; Reis & Renzulli, 1991; 
Renzulli, 1975; Renzulli & Ward 1969, for a detailed discussion about evaluating gifted 
and talented education programs. 14 In addition, see Fetterman, 1988b for a discussion 
about the variety of qualitative approaches to evaluation in education. 15 ) 

Communication and Summing Up 



Writing is part of the analysis process; it clarifies and organizes thought and helps 
to specify ideas and relationships. (See Fetterman, 1989; Hammersley & Atkinson, 
1983.) In some instances evaluators discover in the process of writing that they have 
gaps in their knowledge. If they are still in the process of (or involved in) reviewing a 
gifted and talented education program, they have the opportunity to conduct additional 
interviews and observe specific settings. If they have left the school setting, they have 
only their records and notes to rely on. Frequently embryonic ideas come to maturity 
during writing, as the evaluator is forced to crystallize his or her thinking on a particular 
topic. The reporting of findings usually takes place in memoranda and a final report. An 
interim report may prove a useful tool as well. 

Memoranda 

Memoranda are summaries of the research effort written during various stages of 
the work. They help to synthesize the evaluation and provide a gauge of progress. They 
can also help to consolidate the evaluators' understanding of a situation. They can be 
shared with other evaluation team members and with program participants to solicit their 
feedback. This interactive process of reporting places a check on each evaluator' s 
comprehension of a situation before that understanding is used as a building block to 
comprehend the next stage of development. In addition, it provides other program 
participants with an opportunity to share in the evaluation process and gain a wider view 
of the educational experience. 

Writing memoranda throughout a study also simplifies report writing. The core of 
the report can usually be drawn directly from the memoranda produced throughout the 
study. Virtually all the evaluators have left to do is the final synthesis based on how all 
the memoranda and the responses fit together. In a well-documented evaluation, the final 
report should hold no significant surprises for participants. 

Interim Reports 

Interim reports are preliminary summaries or drafts of the evaluators' knowledge 
up to a prespecified deadline. They are shared with participants and colleagues for 
review. They provide another test of ihe evaluators' understanding of the program or 
culture and provide specific feedback concerning each aspect of the report. As such, they 
can be an invaluable contribution to the quality of the research effort. 
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Final Reports 

Final reports are essential, whether the study is a self-evaluation or an external 
evaluation. A written record puts findings in black and white for all parties to discuss. 
Verbal communications are shaped and reshaped and often lost over time. A written 
record allows participants to return to it to measure progress. 

A final evaluation report is pragmatic. It is likely to have an immediate impact on 
the program studied. The language should, therefore, be clear and concise and free of 
confusing jargon. The body of the final report typically consists of an introduction, 
background, findings, discussion, conclusion, and recommendations. Final reports are 
usually prefaced with an executive summary for policymakers and others who haven't the 
time or the inclination to read the report in its entirety. 

Communication and Program Practice 

Evaluators must communicate their findings and recommendations if they are 
going to be effective. Verbal communications and memoranda issued throughout an 
evaluation keep evaluators and program personnel on track and provide a quality control 
on the data while the evaluation is still in progress. (By the time the final report is 
written, it is often too late to correct erroneous ideas that have been used to build the 
conceptual foundation of the evaluation findings.) Interim reports are valuable forms of 
communication for program participants, sponsors, and evaluators. They create 
benchmarks in the study and provide interested parties with findings in progress, current 
thinking, and a progress report on the evolving direction of the evaluation. All of these 
interim forms of communication minimize surprises, gross errors, and related problems. 
The final report puts a cap on the entire experience for the evaluator, participants, and 
sponsor and typically includes recommendations that, if used properly, will serve as a 
guide to improve program practice. 

Case Study: Hartland 



An annotated case study of an evaluation of a gifted and talented program in 
Hartland (pseudonym) serves to highlight many of the methods, concepts, and techniques 
presented in this discussion. In addition, it demonstrates the scope of evaluation. This 
case study presents the context, including background information about the program 
problem, provides an illustration of a program description, and highlights the role of 
verbatim quotations as a valuable data source and a powerful tool in reporting findings. 

This case study also presents an analysis and assessment of standard gifted and 
talented program mechanisms, specifically referral, identification, and selection criteria 
and procedures. Recommended refinements to these mechanisms are also discussed. The 
case study demonstrates how an analysis of underlying factors may be required. In this 
case, the socioeconomic context of the community provided a useful insight into minority 
representation in the program. The evaluation served as a national test case for gifted and 
talented education and minority enrollment. 

Evaluation periodically addresses larger sociopolitical concerns. The Hartland 
evaluation addressed the issues of equal opportunity, ability, and achievement in 
American education. It also discussed controversial issues such as the quota system. 
Finally the evaluation dramatically illustrated the complexities and pressures associated 
with presenting evaluation findings, reports, and recommendations in the public arena. 
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Importance of Context 

This brief introduction to the evaluation illustrates how evaluation designs are 
shaped. All evaluations are shaped by a specific concern or set of concerns, ranging from 
academic and fiscal accountability to program improvement. The scope of this 
evaluation was shaped by a particular problem — minority enrollment. An evaluation of 
the entire program was needed, and thus many of the specific evaluation steps taken 
would have been needed in any evaluation. However, the evaluation resources were 
focused on addressing the problem. All of the programmatic component reviews were 
conducted in the context of this concern. 

Gifted programs throughout the country share the problem of low minority 
enrollment (see Humphreys 1984; Lemke 1984). The Hartland school district received 
national attention for its failure to enroll a proportional percentage of black children in 
the gifted program in the early 1980s. According to a newspaper, "the gifted and 
talented program is in danger of losing $57,000 in state funding because of a finding [by 
a state study 1 that the program discriminates against minorities' 9 (pp. 3-4). The minority 
enrollment figures did represent a red flag. Blacks represented 40 percent of the third- 
grade population, from which the gifted program participants were drawn. However, 
only 0.4 percent of this population was selected to participate in the program. 

The battle lines on the state political level were clearly drawn. The State Board 
of Education withheld funds because the selection process had not produced proportional 
minority representation in the program. The school board voted to sue the state unless 
the two parties could come to an agreement. In the meantime, the district refused to alter 
its procedures, believing that they were sound and adhered to state policies and 
procedures. Each party sincerely desired to resolve the matter on its own terms. As c 
result of this standoff, the state and the district agreed that an independent evaluation of 
the program would be useful. The state presented a list of evaluators; however, the list 
included individuals who had already concluded in an earlier study that the program 
selection process was seriously flawed. The district wanted to know if a problem existed, 
but at the same time it feared a study with a foregone conclusion, for practical and 
political reasons. A compromise was struck. The district decided to look for an outside 
evaluator who had experience in gifted education and a reputation for fairness, and the 
state agreed to look at the report. The district called Stanford University and requested 
my assistance. 

All evaluations have a focal point Ideally, the evaluator makes this focus explicit 
and determines whether the focal point is directly relevant to the specific problems faced 
by the program at the time as well as to generic programmatic concerns. This is typically 
determined through interviews with key stakeholders, sponsors, policymakers, and 
participants as well as a review of archival material about the program during the first 
phase of an evaluation. 

Program Description 

Program descriptions — a first step in any program evaluation — are drawn from a 
variety of sources and include a brief description of the program and some of its 
educational processes. The program description that follows was developed from 
interviews with students, graduates of the program, teachers, parents, and community 
members on site. In addition, it required observations of and participation in classroom, 
extramural, and home activities. Archival material was also instrumental in building the 
history of the program and identifying corroborating forms of information — triangulating 
data. 
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The Academically Gifted Program serves gifted children from grades four 
through eight Gifted students are drawn from the entire district to study in an all-day 
homogeneous grouping program. Consolidating the gifted program in one building has 
not fostered elitist behavior. One principal described it as a leveling experience for many 
students who've been used to being the top student in everything they do. 

The gifted program began as an experimental program in 1963 and the program 
was adopted on the basis of the evaluation results. The program, which is based on a 
high ability and high achievement enrichment model, has a core curriculum similar to the 
district's curriculum. Additional activities and course requirements supplement the 
gifted program's basic curriculum Special courses, including speed reading, foreign 
language, speech, debate, archaeology, and anthropology are. also offered. Students also 
routinely take part in special interest activities such as intramural sports, computers, 
sewing, drawing, and photography. 

In many respects, the enriched curriculum offers gifted students an educational 
experience that differs qualitatively from that offered their chronological peers in the 
regular school program. Speed reading is provided for seventh-grade students. 
According to the principal and the reading teacher, "Speed itself is not the primary 
goal. 99 Students were observed reading at 800, 900, and 1,200 words per minute. 
However, they learn to vary their speed depending on the text they are reading. 

Foreign language training begins in the fourth grade. The first three years of the 
program stress conversation. During one class, conversational contests were used to 
make foreign language instruction more exciting. Students were given one minute to 
engage in conversations using specified words. The competition required precision in 
comprehension and pronunciation. The program enables students to enter second- or 
third-year foreign language training in high school. It also provides students studying 
French the opportunity to live in France with a French family during the summer. In 
addition to reinforcing linguistic skills, this is a culturally enriching experience. One 
student spoke enthusiastically about her experience in Paris. This portion of the 
program, however, requires parental financial support. 

Speech and debate are also part of the seventh- and eighth-grade curriculum. 
The speech classes require demonstrations that emphasize public speaking skills. For 
example, in one class a student showed how to make French fries. 

Eighth-grade students also have the chance to participate in an archaeological 
dig under the supervision of the archaeological staff from a local university. They use 
metric measurements to record observations and findings at the site. Students plot the 
artifacts on graphs and sift the soil through half-inch mesh screen to catch pottery shards 
and other fragments. Classroom preparation for the field school is required Courses 
include archaeology, anthropology, and time measurement. In one class, students 
measured and plotted simulated archaeological findings to prepare for the field school. 
Students worked in teams, cross-checking each other's work and striving for accuracy. 
In addition, they take an introductory research course that orients them to the scientific 
approach, including research concepts and techniques, and that culminates in a research 
paper. Students in the gifted program have made the school a successful competitor in 
the Junior Academy of Science State Expositions. 

On-site observations and interviews documented a match between program goals 
and actual performance, including engaging and challenging gifted and talented students 
in a high ability and high achievement model program, preparing students for 
participation in a foreign language immersion experience, preparing students to enter 
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second- and third-year language training in high school, preparing students for and 
plScipating in an archeolSgical dig, and competing in the Junior Academic Science State 
Exposition. 

Verbatim Quotations 

Data collection is an ongoing process in an evaluation. A valuable and often 
overlooked form of data is the verbatim quotation. They provide the insiders 
perspectives in their own language. In an evaluation report they allow the reader or 
reviewer of the evaluation to look at some of the raw data. Verbatim quotations from 
progZ participants have a tremendous face validity. These data are strengthened by 
cross-checking and comparing the data with other sources. However ^rbattm 
quotations furnish a valuable data source for analysis and, as part of a report, bring the 
Stop's findings alive. The following excerpt illustrates effective use of verbatim 
quoSns in an evaluation report, reviewing educational leadership in the program, 
teacher and staff member commitment, curriculum, and program climate. 

77* evaluation found that the principal provides strong educational leadership 
and is respected by program teachers, students, and parents. He is also viewed as a 
supportive individual. As one parent explained: 

I think John is an asset for this school. He is very pro students and he is good for 
this prosram. And to me, what impressed me was something my daughter said. 
She said she was just impressed they could just talk to him just, you know, ike an 
average person. She said, "Mom, I could never do that at the other school! I 
thought that was important for them to feel that they could goto John for 
anything. You're very apprehensive about what you do in school but if you have 
that fear gone, you feel you are comfortable. 

The teachers in this program are dedicated and innovative educators, sensitive to 
the needs of sifted children. One seventh-grade student explained that these teachers 
"k^m^L and they know a better way to teach it [the subject matter] so it 's moslyfun 
andZteTesting. " An award-winning math teacher in the program provided an insight 
Zto this student's comments: "I give.Jots of brain-teaser kinds of problems. I d say 50 
percent of the kids will devour those and ask for more....Once you get a kid to realize he 
has nrnch greater potential than he ever imagined-then you've got him. Teachers 
teiT^^fftotructian that address individual student differences . They often 
finda sharp contrL between teaching in the regular school program ^m^fied 
program, as one teacher remarked: "When 1 first started teaching [ in the district] it was 
alZstlike pulling teeth. And now [in the gifted program], I'm almost running to stay in 
front of them" 

The curriculum for gifted students is presented in a coherent and consistent 
fashion. One parent who has a number of her children in the program S^e her 
TxpLation for this success: "The reason I think it is a strong program is because they 
have clear well-defined goals and their selection process is directly related to those 
Tl " Or*ZzaZnal Mils are reinforced in all classrooms. Qualities such as self- 
^ted^eSSidual responsibility are encouraged, and teachers attempt to instill 
a?eZ^oTlcmcemin their students. Asone teacher expand, "I would hope that 
Vwo7dd%ve them some re al values to use right along with their tremendous talents, so 
tZlhey are ofToZ benefit to mankind, not a detriment. " Students are enthusiastic and 
competitive learners, as well as peer educators and resources. 
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The school climate is conducive to the pursuit of advanced education. Like most 
gifted programs, this program had few discipline problems and no graffiti on the school 
walls. Students take pride in their school. One graduate of the program, currently a 
senior in college, shared his feelings in a letter to the principal: 

Today I received a letter from home that included an article about [the 
Gifted Program]. It was a fascinating piece to read— I wish I could be back at the 
program. It really made me think about what sort of benefits I got from the 
district. 

I liked being in the gifted program —regular school was boring. It's hard 
to forget the experiences of third grade at my [regular] home school. For 
example, if the class was working on homework and a student finished his work 
early, he was to select a book from the class library, read it, and write a book 
report on it. While I like reading books and didn y t mind too much having to write 
the reports, after a while it became almost punitive to have to read and have to 
write just because I had worked too fast. That "enforced privilege " became an 
incentive to slow down and try to waste some tune. There was another incident 
when my "Great Books'* group returned to class to find ourselves placed on a 
panel to answer history questions thought up by other students while we were 
away. There *s nothing wrong with a pop quiz, but being expected to show 
publicly how well we knew or didn 't know history was both terrifying and 
embarrassing. That sort of situation leads to an attitude on the part of the other 
students (and even the teacher) of "You're so smart - so you tell us the answers?" 
That year our classes were composed of students of all ability levels, which I think 
was a terrible mistake. Third graders were supposed to learn the multiplication 
tables up to ten, but my group didn 't even make it halfway through. My family 
had to teach them to me during the summer between third and fourth grade, when 
I entered the gifted program. 

Entering the program was hard— a friend from [his home school] took my 
decision to go as a personal attack on her. At [my home school] we faced a lot of 
hostility on the playground from the other students. We were always "different" 
and "outsiders. 99 Switching to [the Gifted Program] was great because we could 
finally enjoy school The teachers were interested in us, and there was no peer 
pressure to slow down and be average — in fact, doing well was encouraged and 
respected 

I really envy the kids who go to [the program] today. They get to work 
with a microcomputer, learn archaeology at a real "dig, 99 and benefit from the 
added experience of the teachers who have worked in the program for a while. 
The chance to learn a foreign language in grade school has been invaluable in 
more ways than one. Mastering grammatical terms and structures then has 
helped immensely in writing papers now. Even if my theme isn y t clear, at least the 
form is great! I've been studying French in college, and it seems incredibly easy 
after German. Special programs like foreign language were both fun and 
rewarding. 

I noticed that the class of [recent graduates] interviewed by the local 
newspaper mentioned [the Program's] freedom quite frequently as having been 
important to them. It was important to me, too. Being treated as a responsible 
person was a great privilege that I didn *t appreciate until high school I never 
did get used to showing hall passes to security guards or being a "number" in a 
PE class. 
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Students are encouraged to explore their ow n interests on the same intellectual 
plane. As one parent who had one child in the program and one in the regular school 
system elaborated: 

I think the kids get more encouragement here [in the gifted program]. To 
achieve maybe higher than where they are. You know, I think they have the 
attitude that okay, the child may look and say this is difficult but 7 think the 
teachers encourage them to try it anyway. I think they have the freedom — more 
freedom to do it here because of the level; they don 't have to split their time 
between the child who may be just, you know, not so much a slow learner, but he 
just is not at this person 's level. You 're holding back and so this way I think 
everybody can almost work at their own level without holding back someone else. 
That's what I like about the program 

This facet of the program is also appreciated by students. One seventh-grade 
student compared the program with her old school: 

[At the old school] some kids are faster and they get their work done 
better and some of them are slower and some couldn 9 t do it and you had to wait 
for thenu...You just had to sit there and be quiet....This school is better. You 
switch classes — which means you have a teacher that specializes in special 
subjects so they can spend more time on that subject. They don 't spend 15 
minutes on math and you just do the assignment. You are spending a whole 45 
minute period on one subject. So I understand it better. 

A number of students display exceptional talent through individual efforts. For 
example one thirteen-year-old student's software programming accomplishments for a 
Silicon Valley firm have been publicized on ABCs "Good Morning, America" Lhow. 
Group efforts are also common. School spirit and camaraderie are evident in group 
classroom projects and extracurricular activities. Many of the gifted children in the 
program are also successful athletes in various sports. 

Concern for the student's development extends outside the classroom. One parent 
explained how "these teachers are very supportive of sports activities, " as evidenced by 
their routine attendance at both scrimmages and official games. One parent describes 
the degree of parental involvement in the support system of the program: 

Interviewer: Do you notice a difference [from the regular school system] in terms 
of parental involvement in the gifted program? 

Parent: (laughter) TItat's an understatement. You should have been here last 
night... [at a parent's night]. It was standing room only. There were over 300 
parents. Okay, now this week at my children 's other [ regular] school, I will go 
and there will be maybe 100 parents. I mean it's definitely that much of a 
difference. 

Parents share their appreciation of the Gifted Program with school 
administrators at graduation. In one letter, a parent wrote the following to the principal: 

We wish to express our appreciation of the support given to [our 
daughter] by you and the staff. Contrast the happy, excited, and confident 
teenager who completed eighth grade last week with the discouraged and 
despondent fifth grader who was the subject of intense peer harassment and 
derision three years ago and you can understand our appreciation. Your 
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judicious actions and the staff's interested involvemi n 4 undoubtedly had much to 
do with making [our daughter's] elementary years rewarding. She was fortunate 
to have the type of challenge, guidance, encouragement, and opportunity for 
independent development that the program provides. For all these professional 
benefits we are truly thankful. 

Another parent's letter communicated similar sentiments: 

In a nutshell, [the Gifted Program] has let the sun shine in. My daughter 
always liked school, always did well, always seemed content. Now, however, 
school is pure joy. Tlie change in her is striking and only proves to me how 
unwittingly we ignore the needs and potential of many students because they make 
it so easy for us to do so. In fact, we are no doubt allowing untold numbers of 
youngsters to slide along at a reduced pace because we are unwilling to raise our 
expectations. If I could have one educational wish..., it would be that more of 
them could experience the sunshine that my daughter has found. Thanks for a 
super program. 

The verbatim quotations in this selection provide an insight into the relationship 
between the principal, teachers, students, and parents. In addition, they show support for 
stated program goals and objectives, including committed staff members, involved 
parents, and a school climate conducive to advanced education. Retrospective and 
reflective verbatim quotations from a graduate of the program are useful in many 
respects. A testimonial from a graduate of the program is a positive indicator of program 
impact. (It is only one piece of data. However, as part of a cluster of positive indicators 
emerging from a variety of sources, it contributes to a strong and compelling finding.) In 
addition, a program graduate can provide additional information by comparing the 
program with the regular system, enabling the reader or reviewer to view the program in a 
larger context. Similarly, a parent with one child in the gifted program and another in the 
regular program can provide excellent comparative insights into the program. Such 
comparisons demonstrate the full value of teacher commitment and parental involvement 
in the gifted program. 

Assessment and Analysis 

Evaluations should document program elements that are working effectively, as 
well as those that are ineffective or counter-productive. The evaluation of the gifted and 
talented program concluded that the district had designed a gifted education instructional 
program in response to the educational needs of students with both general intellectual 
ability and specific aptitude. In addition, the program's strength and community support 
were unquestionable. However, evaluations rarely find a program in perfect working 
order. In fact, typically evaluations err on the side of reporting problems that need to be 
addressed. This evaluation identified and documented numerous problems. The district 
did have areas that needed innovation and improvement Its service to gifted children 
between junior high school and college needed to be strengthened. The evaluation also 
determined that underachieving gifted children were not served and recommended 
extending the program to include those students. It also recommended a mentor program 
and emphasized the need to collect and maintain follow-up data on graduates. These 
findings were based on interviews, observations, and a review of testing data in the 
district. 
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The Mechanisms 

Once the value of the program had been established, an analysis of the referral, 
identification, and selection mechanisms was warranted. These are classic elements of 
any gifted and talented program evaluation that merit attention. This facet of an 
evaluation requires interviews with principals, gifted coordinators, school psychologists, 
and parents. In addition, a review of the documentation associated with the referral, 
identification, and selection process is required, including individual scoring sheets, 
students files, and related district records. This section of the case study also highlights 
the value of outside experts concerning technical issues. 

The mechanisms themselves were found to be standard. On the basis of 
districtwide intelligence and achievement test results and teacher and principal 
nominations, students were referred to a neighboring university for individual 
intelligence testing. A selection committee screens candidates through a predetermined 
formula consisting of the Wechsler Intelligence Scales for Children-Revised (WISC-R) 
(39.65 percent), the Cognitive Abilities Test (CAT) (19.07 percent), the SRA Achievement 
Series (SRA) (16.09 percent), grades (15.14 percent), narrative (5.63 percent), and the 
Characteristics Rating Form (4.42 percent). Students are ranked according to an overall 
score based on this formula. The top sixty students are selected for participation in the 
gifted program. Periodically, students are selected to fill vacancies at various grade 
levels. 

The evaluation found that the Hartland School District is adhering to the State 
Board of Education s rules, regulations, and guidelines governing the identification and 
selection process of gifted education reimbursement programs. Criteria for selection 
have been described in detail and consistently applied to children in the local educational 
agency population. 

The district exceeds the state standards regarding the use of identification 
devices. The state requires a minimum of three identification devices and lists suggested 
methods for the district to use in identifying gifted children. The district has selected five 
methods from the state's list: the WISC-R (intelligence test), the SRA (achievement test), 
' teacher recommendation, past grades (past performance), and individual rating sheets. 
In addition, the CAT is used. Identification criteria are established before students are 
selected for the program, and specific cut-off scores are adopted when standardized tests 
are used. A direct relationship exists between the criteria for selection and the 
instructional program for gifted students. 

All gifted program teachers are certified and are required to meet two of the three 
state requirements. They must all have attended a summer training institute approved by 
the Office of the Superintendent of Public Instruction for teachers of the gifted, and have 
had at least two years (some as many as eighteen years) of experience working with 
gifted programs. This component of the evaluation shifts from process and outcome 
description to accountability in an almost audit compliance fashion, highlighting the 
multifaceted nature of program evaluation. 

Refinements 

Evaluations, particularly formative evaluations, 16 are designed to improve the 
program during the evaluation. This is particularly important in cases where a significant 
problem has already been identified and is the focus of the evaluation. In this evaluation, 
numerous refinements or recommendations were made in memoranda throughout the 
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evaluation and compiled (with school district responses) in the final report. Some of the 
recommendations to refine the progrc.m operation follow. 

Although the district met or surpassed state requirements, a review of the specific 
referral, identification, and selection mechanisms in practice suggested that refinements 
were needed. Viewing each mechanism as a gatekeeping function highlighted the 
significance of each problem and the nature of the necessary refinement. 

For a student, referral is the first door into the gifted program The district has 
three referral pools. The first is composed of third- grade students who score at the 80th 
percentile or above on districtwide CAT and SRA tests. The second pool consists of 
students who may have missed one of the tests by being absent, and the third pool 
includes individual teacher or principal nominations. Referral statistics indicated that 
only nine black students were referred to the district gifted program—four from the first 
pool and five from the third. Only two students were referred from five predominantly 
black schools in the district. One black parent explained, "I have heard from others that 
sometimes the teachers don't recommend the students, and it's not so much that it's 
because they are black, but...that some of the principals want to have those high 
achievers at their schools. " In other cases, the parent explained that low teacher 
expectations were a problem: 

The teachers walk into the school. Nothing is expected of them. They 
walk in and say, "All right, look at this address, where is he from? Okay, I know 
he is not going to be able to do this. Vm not going to spend that much time if he 
does not get it right away; I'm not going to spend that much time with him. Label 
him learning disabled or slow learner.... " It's just so sad they feel like no one 
wants to teach down there. "Oh, it's boring, I can 't teach those kids. " They just 
feel they can't learn, but if they noticed, if they give these kids a little extra 
attention, a little encouragement, they would do well. 

The number of minority students in the program can increase only if the pools of 
minority applicants increase. The evaluation pointed to the problem of teacher 
expectations and the need to conduct in-service training programs in identification and 
referral procedures, particularly in schools that had not referred any students to the 
gifted program. The evaluation recommended that teachers with the greatest predictive 
ability (on the basis of past performance) should help develop and conduct in-service 
training programs. Although it would not guarantee increased selection, such a program 
would be a first step toward refining the district's procedures and instruments. 
Increasing the pool of potentially qualified applicants is a necessary yet insufficient 
baseline. 

Identification procedures are a second door to the program. Clark ( 1979) and 
Getzels and Dillon (1973), among others, emphasize the role of identification methods in 
explaining underrepresentation. The gifted and talented program had to make several 
modifications to answer political, pragmatic, and technical concerns. The selection 
committee members responsible for reviewing student tests and documents all belonged 
to the same school. Other schools had no representative on the committee to serve as a 
potential quality control and political advocate. The evaluation recommended that the 
district consider appointing a representative to the committee from the southern, 
predominately minority schools. 

A second problem involved the Characteristics Rating Form, one of the 
instruments to identify gifted children in the district. The form was out of date and lacked 
internal consistency. For example, some questions juxtaposed a "poor" rating response 
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with a "better than a good many" rating response. An updated list of behavioral 
characteristics was shared with the district to enable it to modify and improve the 
selection process. (See Davis & Rimm, 1985; Renzulli & Hartman, 1971; Tuttle & 
Becker, 1 980 for a useful collection of behavioral characteristics checklists.) 

A third problem concerned the districtwide tests. The WISC-R, SRA, and CAT are 
acceptable and appropriate tests for a gifted program oriented toward high ability and 
high achievement. (See Hagen, 1980, regarding the value of the WISC-R and the CAT.) 
A review of the district achievement test score sheets revealed no significant problem, but 
a minor mechanical problem did emerge. The pencil quality of recorded answers was 
inconsistent, which suggested that optical scoring might be affected. Such an observation 
seems trivial, but in a similar case the consequences were significant (Breckenridge 
1984); 

Hillsborough County, Florida, public-school officials have pinpointed 
faulty pencils as the reason a computer misread 10,500 answer sheets for a basic- 
skills test taken by 85,000 students earlier this year. 

Administrators estimate it will cost $40,000 and take 26 days to re score 
all 85,000 tests, (p. 7) 

In an investigative-evaluation of this type, all levels, including abstract and 
mechanical, must be explored. Low-graphite pencils may be used disproportionately by 
one segment of the population. In addition, sections of individual (SRA) score sheets 
were completely blank. The evaluation emphasized that this finding indicated a need for 
additional analysis. Further study would enable any school district to determine the 
frequency of the problem and any correlation with specific teachers, schools, or 
populations. 

A more substantive problem involved the CAT. One of the cut-off scores for the 
first pool of students (Category I referral) was based on the CAT The lowest score was 
selected from the verbal, nonverbal, and quantitative elements of the test. The evaluation 
demonstrated how a higher measure of consistency could be achieved by selecting a 
single element for all students, assuming students meet an established standard on each 
of the CAT subtests. (The quantitative subtest appears to be the best predictor at the 
middle and secondary school levels.) This refinement may improve the accuracy of the 
match between the type of student (verbal, nonverbal, and/or quantitative ability) and the 
program curriculum. 

As Hagen (personal communication 1985) points out, "It is extremely important 
that the selection process for gifted programs should be closely related to the cognitive, 
academic, and other demands that the program makes on students. " In addition, the 
evaluation recommended the use of raw scores or scaled scores rather than percentiles in 
determining this facet of a student's eligibility for the program. Score averaging was 
also discussed Hagen (personal communication 1985) strongly recommends that 
"Standard Age scores be used for selection, particularly if the selection procedure 
involves averaging two or more of the scores. Whenever averages of two or more of the 
test scores are used for selection, it is important to remember that a distribution of 
averages is typically less variable than the scores used to compute the average. " The 
evaluation also noted one deviation from the conventional administration of the WISC-R; 
Students did not receive the vocabulary section of the test, which is the greatest predictor 
of achievement. Although this practice does not compromise the test validity or 
reliability because the four remaining subtests are averaged (see Wechsler 1 974, 
Appendix E, p. 190), it does represent a different approach. Before the evaluation was 
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completed, the district research director requested that the vocabulary section be 
administered in the future. 

A more technical recommendation involved the weighting system. Vie following 
weights were given to each variable in determining the ranking score of the students: 
WISC-R, 39.65 percent; CAT, 19.07 percent; SRA, 16.09 percent; grades, 15.14 percent; 
narrative 5.63 percent; and Characteristics Rating Form, 4.42 percent. Straight 
percentages, however, can give additional weight to IQ scores. 

The evaluation suggested a refinement to improve the accuracy of the weighting 
mechanism. Briefly, instead of using straight percentages for each variable, percentages 
could be divided by the standard deviation of the group. This refinement would further 
increase the accuracy of the measurement by distributing these student scores on the t 
basis of the gifted population scores (for further details, see Guilford, 1956). The 
evaluation also suggested that SRA raw scores could be used instead of percentiles. 
Adopting growth-scale values would at least represent an improvement on the use of 
percentiles. Raw scores can convert to normal curve equivalents (NCEs) to increase the 
accuracy of the measurement. 

A final recommendation involved rank-ordering students eligible for a 
replacement slot. Periodically, students left the program, either by choice or necessity. 
Potential replacements, however, were not sequentially ranked according to their total 
scores. This practice, which was unfair to students and their parents and inconsistent 
with program procedures, was also a significant political liability. Politically pressured 
district administrators may have appreciated the latitude or discretion this loophole 
offered, but the small amount of discretionary power they gained was insignificant 
compared with the charges of preferential treatment they face. 

Overall, the evaluation concluded that district referral, identification, and selection 
systems were appropriate given the program's model of high ability and high 
achievement. Suggestions and refinements to improve the accuracy of the existing 
system were often geared toward enhancing the probability of increased minority 
representation. However, these fundamental mechanisms clearly were not the most 
significant cause of low minority representation in the program. The major underlying 
factors lay in the community, not in the school. 

Analyzing Underlying Factors: Socioeconomic Context of Minority Representation 

Evaluators need to persevere when a review of standard program components fails 
to provide a useful insight into program operations. Evaluators should exhaust all useful 
and realistic data sources whenever possible. This excerpt emphasizes the need to look 
outside the school and into the community to understand the larger dynamics operating in 
a gifted and talented program. It also highlights the value of archival data such as census 
data to meaningfully interpret the enrollment data in context. This portion of the 
evaluation also relied on interviews with educational administrators, teachers, and black, 
white, and Asian parents in the community. Memoranda on this topic were useful to 
validate my preliminary findings with major stakeholders in the district and the 
community. 

The state was primarily concerned about black student representation in the 
district's gifted program. Pertinent socioeconomic data were used to explain the 
disproportionately low black student enrollment in the program 
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In the district, white students comprised 60% of the third grade and 55 of 737 
white students (7.5%) were selected to participate in the gifted and talented program. 
Given their size in the school population, Asians were the most over-represented group in 
the program. Asians constituted 2% of the third-grade enrollment. However, in 1984, 3 
of 26 Asian students (12%) were selected to participate in the gifted program. In 
contrast, 2 of 475 black students (.4%) were selected to participate. Clearly, blacks were 
statistically underrepresented in the program. Similar patterns of under- and 
overrepresentation, resulting from social variables, are common in gifted programs. 

Like most communities, Hartland is not socioeconomicaUy homogeneous. 
Approximately 72% of the black population lives in the poorer, southern section of town. 
The southern section has by far the highest unemployment rate, the highest rate of female 
heads of household with children and, conversely, the lowest rate of husband-wife 
households. The southern section has the lowest income and educational achievement 
levels in the community. It also has the highest percent of renters and the highest rate of 
vacant commercial units. Neighborhood housing has severely deteriorated. 

In contrast, the middle and northern sections of Hartland are predominately white 
(74%). These sections of the community enjoy average to very low unemployment rates 
and the highest rate of husband-wife households. They also have the highest incomes and 
educational levels in the community. The middle and northern sections of the town have 
the highest percentages of home owners, few vacant commercial units, and neighborhood 
housing in good to excellent conditions. 

According to the 1980 U.S. census (U.S. Department of Commerce, issued in 
1983), the median income for whites was $19,192 and $12,063 for blacks. Thirty-four 
percent of working whites were employed in technical, sales, and administrative support 
occupations; 23% of employed blacks worked at this socioeconomic level. As one 
proceeded up the employment ladder, the discrepancy between blacks and whites 
increased. Twenty-eight percent of employed whites had jobs in managerial and 
professional specialty occupations— 16% in the professional category. In contrast, only 
13% of employed blacks worked in this economic group — 9% in the professional fields. 
Similarly, 29.5% of the black population lived below the poverty line, compared with 
5.6% of the white population. 

Gifted enrollment statistics are a product of societal forces outside the classroom. 
These socioeconomic variables strongly influence this gifted program's enrollment 
statistics (see Barbe, 1956, regarding socioeconomic variables). Given such significant 
socioeconomic differences in Hartland, disproportionate representation in a high 
ability/high achievement program is not surprising. A plethora of social forces inhibit 
proper academic preparation of blacks in Hartland, ranging from low incomes to 
insufficient educational background Conversely, white families in Hartland have the 
advantage of higher incomes and better educational backgrounds than blacks. 

The evaluation had to take the sociocultural variables into account when 
evaluating program enrollment statistics. In an ideal world in which all students share 
the advantages of a supportive, enriched background, IQ scores would have more 
meaning and proportional representation could be expected. Hartland' s vast 
socioeconomic differences, however, make such expectations unrealistic. 

Addressing Larger Sociopolitical Concerns: Quota Systems and Equality of Results 

Evaluations often address larger social and political concerns beyond the program. 
They should be considered secondary or supplementary to a focus on the program. 
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Larger issues should not consume more resources or space in the report than program 
concerns. However, they should not be neglected when relevant. This evaluation 
highlighted a problem common to many gifted and talented programs in the United States 
and had direct programmatic implications. This portion of the evaluation required 
experience in the field, knowledge of the equity literature, and knowledge about the State 
Board of Education's position — based on governmental archival material, interviews with 
government officials, and newspaper articles, as well as interviews with program 
participants. 

Problems of unequal representation are often addressed by imposing a quota, 
making proportional enrollment a legislative process. In Hartland, the district faces 
State Board of Education pressures to establish greater minority representation in its 
gifted program. A quota system would produce proportional representation and be 
politically expedient. However, a simple percentage quota system— unrelated to existing 
criteria— would have significant and potentially deleterious effects on the program and 
on district enrollment. 

Hartland's program is designed for high-ability and high-achieving students. 
This model serves motivated, goal- and achievement-oriented students with high 
intellectual potential Funds limited the program to sixty students. Such fiscal 
constraints already cause a loss of district enrollment; many eligible students (not 
included in the sixty selected) are lost to private schools. 

The larger mission of the gifted program is also at issue. Hartland* s program is, 
by its nature, designed to meet the needs of children at the tnargins of intellectual 
distribution. Implementing a quota system redefines the term gifted and equates equal 
opportunity with equal ability — an egalitarian fallacy. Applying a simple percentage 
quota system, unrelated to existing selection criteria, to a gifted program makes no more 
sense than applying such a system to an athletic team. The purpose of the program in 
Hartland is to select and serve children with specific intellectual and motivational needs 
and qualities. Students having the highest predictive criteria performance should be 
selected. These criteria for program selection should be based on statistical measures 
supplemented by more subjective measures. 

At the same time, scores must be viewed realistically. The difference between an 
IQ score of 130 and one of 128 is academic. The child who scores 128 will, in all 
probability, perform as well as the student who scores 130. Comparing scores is much 
like comparing stereo specifications. The difference between a receiver that has a range 
of 20 to 20,000 MHz and a receiver that has a range of 17 to 25,000 MHz is extremely 
small, given that the human ear can only hear between 20 and 20,000 MHz. Similarly, a 
quota system that selects for minority status within an acceptably high ability and 
achievement range need not require perfect numerical comparability with competing 
students. Such an approach would not undermine the program. 

The faulty equation of equal opportunity with equal ability is followed by an 
equally misleading equation —that of equal opportunity with equal achievement or 
results. 

One intellectually important consequence of the Coleman inquiry into 
educational inequalities was that concepts of equality began to polarize around 
two dominant principles: one was the old traditional value of equality of 
opportunity, but the other was the newly appreciated— if not newly conceived— 
idea of equality of results....as Coleman clearly perceived, when equal results 
were achieved in academic records it did not follow as a matter of course that 
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they sprang from equality of conditions in the schools. Beyond this difficulty, the 
somewhat crude results thrown up by test scores and other school records could 
too easily be transmuted from methods of measurement into educational aims. 
When the aim of the schools was to achieve an equality of recorded measurement 
the system might be in working order, but some doubt would arise as to whether it 
was a system of education. Educational specialists would find nothing to surprise 
them in this problem Its most acute form had long afflicted America 's more 
gifted children, frequently held back from anything like their full potentialities by 
the stubborn pace of satisfied mediocrity. When quality meant equal opportunity 
for each child to develop fully his or her own potential, it could not be easily 
reconciled with the view of equality which aimed to produce a steady stream of 
similar products and failed to offer the incentive, the equipment, or the 
intelligence needed by children of innately superior abilities. (Pole, 1978, p. 352) 

The issues of equal opportunity, ability, and achievement or results can have a 
profound effect on the operation of gifted education programs. These basic philosophical 
concerns, as well as pragmatic and political concerns, must be addressed before any 
alteration of a program's existing model is undertaken. 

Evaluation Findings, Reports, and Recommendations and the Public 

The test of an evaluation is often in the findings and recommendations — which 
are presented in the form of a report. Highly publicized evaluation findings and 
recommendations adds a qualitatively different level of difficulty to the effort 
Difficulties can be minimized by following normal evaluation procedures including 
cross-checking findings and sharing preliminary findings with key stakeholders with 
memoranda and interim reports. The press operates independently of an evaluation. 
However, they can pressure program officials to implement recommendations and ensure 
continued accountability. 

The evaluation's findings were presented to the school board in a controversial 
and highly publicized atmosphere, reflecting the political tensions that ran through the 
entire study. Three television channels and a handful of newspaper and radio reporters 
covered the affair. Cameras were rolling; floodlights and microphones were everywhere. 
The board.heard a point-by-point report and then explored specific points in greater 
detail and asked for additional suggestions. At the subsequent press conference, some 
reporters pressed for a vindication of the city. Others viewed this occasion as an 
opportunity to strengthen the state's case. 

Predictably, television coverage presented as many interpretations of the 
evaluation report as there were reporters. One anchorman reported that the Stanford 
researcher had given the program a "gold seal. " Another anchorwoman reported 
"mixed findings" by the Stanford professor. A third reporter emphasized the 
socioeconomic factors discussed in the report. One paper emphasized the evaluation 
recommendations regarding the weighting system for tests to select students and the 
warning that teachers too often have low expectations of black children. This variation 
in interpretation is one of the reasons I emphasize the need to produce an evaluation 
report. Verbal communications are often lost or modified intentionally and 
unintentionally. In addition, a report enables everyone to have the same information. It 
also enables everyone to return to the written word in case of disputes. 

For its part, the district considered each finding and recommendation on its own 
merits. The district made more than twenty-three program changes based on the 
evaluation recommendations. 
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Some of the changes include having a representative from schools, whidn * 
have a higher percentage of blacks on the selection committee for the progratti; 
training teachers in schools where few or no students have been referred tQ.4he 
program to recognize gifted students; establishing quality controls on the system 
used to select students; and developing a more definitive process for ranking 
students for selection. 

The board president sent a copy of the report and program changes to the state 
superintendent of education to resolve Hartland's ongoing conflict. The district agreed 
to increase the pool of minority applicants. However, the director of research recognized 
that "increasing the pool of minority students in the referral group will not necessarily 
increase the number of minorities selected as long as the program is maintained at its 
original intent and present purpose. " 

According to the district's research director, "they wanted an academically gifted 
program for children with high ability who have proven academic achievement." Each 
community selects the type of program it believes to be most advantageous for its 
children. As Whitmore (1980) points out, "Methods of identification are determined by 
the goals of the program for which students are being selected 1 ' (p. 19). Hartland's 
model is highly selective and is only one of many excellent gifted and talented program 
models. 

Some programs have broken Hartland's criteria into separate parts for selection 
consideration: high achievement for one group and high ability for another. Using the 
high-achievement category alone and lowering IQ standards for economically 
disenfranchised students has been successful in increasing minority enrollment in other 
districts. However, a different kind of program is needed to accommodate students 
selected under these arrangements. Moreover, some difficulties with retention have 
arisen with this selection system. A program that places greater emphasis on talents and 
creativity than on academic giftedness would probably increase minority representation. 
Such a focus would, however, require redesigning the program structure and curriculum 
as well as staff configuration. 

The Hartland program evaluation was a test case for a basic problem in gifted 
education programs throughout the country. The evaluation of Hartland's program 
demonstrated that problems in the classroom often spring from the local community (see 
also Tannenbaum 1983, p. 353). Program and community variables must be thoroughly 
evaluated before conclusions about a district are made. Since schools often reflect 
societal forces, inequities in the community will create inequities in its schools. To 
accuse the schools and school programs of being the sole cause of such problems is an 
example of blaming the victim (see Fetterman, 1981). 

All useful evaluations will have a focal point, a set of goals, outcomes, or 
concerns that shape the effort. Most evaluations will require the type of detailed program 
description provided in this presentation, as well as a description of the program 
processes and effects. Many will confront technical issues. Few evaluations will require 
such a thorough examination of larger social and political issues in education. However, 
all evaluations have the potential of grappling with controversial issues of this scope and 
significance and in the public forum. 
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Concluding Thoughts 



Evaluation is essential to learn how a gifted program works, how effective 
programs are, and how to raise their standards of quality. Self-evaluations should be a 
routine part of daily program activity. Students, teachers, administrators, and parents 
should be encouraged to conduct informal self-appraisals on a daily or at least weekly 
basis, questioning and comparing what students are doing in relation to stated program 
goals and objectives. Systems should be developed and implemented to give regular 
feedback to students, teachers, administrators, and parents, including parent-teacher 
conferences, faculty meetings, and student performance conferences. 

External and independent evaluations complement self-evaluations by ensuring a 
more objective and credible appraisal. Formative evaluations provide a continual flow of 
information to program officials throughout a review to improve program practice. 
Summative 17 evaluations can enhance formative evaluations by providing additional 
knowledge with a focus on policy decision making. External evaluations, whether 
qualitative or quantitative, formative or summative can improve program practice and 
student performance. Independent evaluations also help to establish the utility of such 
approaches in gifted education as acceleration, enrichment, and special group settings. 
They are also more credible to sponsors and outside agencies, particularly concerning 
sensitive or controversial issues. 

Nearly all programs can be improved by a critical review. If unexamined, the 
health and well-being of gifted and talented children and the future of the nation are at 
risk. Together, these approaches play an essential part in the development, maintenance, 
and understanding of educational programs for the gifted and talented. 
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General Evaluation Guidelines 



Guideline One: Make sure the evaluation serves the practical information needed 
by the targeted audiences. 

Take time to identify who the key stakeholders are including administrators, parents, 
teachers, and students. Find out what they want to know and then develop a plan to 
address the most salient concerns. 

Guideline Two: Make sure the evaluation is realistic (politically and pragmatically) 
and cost effective. 

There is no point in conducting an evaluation if there is no political support for a program 
or if there is good reason to believe the findings will be hidden or altered. Care must also 
be taken to ensure that an evaluation effort does not significantly drain program 
resources. 

Guideline Three: Make sure the evaluation is conducted in an ethical manner. 

The rights of program participants must be protected. Care should be taken to ensure 
privacy, freedom of information, and confidentiality (if promised). 

Guideline Four: Make sure the evaluation is as accurate as possible. 

Take the time to seek out various sources of information and cross-check data sources. 
Double check all figures and interview notes. Judgments about the data should be logical 
and reviewed by independent sources whenever possible. 

For a more detailed discussion of evaluation standards and guidelines see The Program 
Evaluation Standards by the Joint Committee on Standards for Educational Evaluation 
(1992) and Rossi's Standards for Evaluation Practice (1982). 

Specific Guidelines for Evaluating Gifted and Talented 
Education Programs 

Guideline One: Make sure program documentation exists. 

Program documentation should describe the program's philosophy; curriculum; staffing; 
financial, library, and computer resources; identification and screening procedures; and 
selection criteria. In addition, classroom schedules and maps of the physical layout will 
facilitate any evaluation. 

Guideline Two: Make sure you review as many relevant data sources as possible. 

Interviews and observations are critical. In addition, archival documentation, such as 
newsletters, financial reports, student letters, parent letters, past evaluation reports, 
newspaper articles, and many other documents provide pertinent data about the program's 
impact and role in the community. 
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Guideline Three: Make sure you compare the program's stated goals with their 
actual performance. 

Does the program operate in accordance with its own philosophy (academically and in 
terms of governance)? Does the curriculum reflect the philosophy and goals of the 
school? Do the staff members appear to understand and implement the stated program 
philosophy? How do teachers translate the program's philosophy into practice in their 
teaching? 

Guideline Four: Make sure you describe and assess the climate* 

Are students engaged? Are teachers stimulating, thoughtful and knowledgeable? Is 
communication between staff and administration constructive and cooperative or 
antagonistic and fragmented? Similarly, what is the nature and tone of communication 
with and among students. 

Guideline Five: Make sure you talk to students* 

The purpose of gifted and talented education programs is to serve students. Time should 
be devoted to informally interview students about their own progress (including a review 
of their portfolio, records, or projects) and their evaluation of the program. Student 
academic achievement and behavior code data are critical to any gifted and talented 
education program evaluation, i 

Guideline Six: Make sure program finances are reviewed* 

Is the program budget sufficient, if not, why not? Is the money being used as intended, if 
not, why not? Is financial program planning adequate and appropriate to meet the needs 
of the program in the foreseeable future? 

Guideline Seven: Make sure community and school board components are included 
in the evaluation. 

Do community and school board members support the program? What is the evidence? 
Do parents participate in the program? What are the obstacles to community and board 
support if any? 
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Notes 



1 This discussion is informed by the definition of giftedness in the Jacob K. Javits 
Gifted and Talented Students Education Act of 1°88. 

2 During my presidency of the American Evaluation Association, I created an 
evaluation approach called empowerment evaluation. This approach involves using 
evaluation to help others help themselves. One facet involves encouraging program 
participants to conduct their own evaluations by sharing useful models and concepts that 
help them look at themselves critically. 

3 Triangulation is both a conceptual orientation to guide inquiry and a technique 
for testing data in the field. 

4 Construct and predictive validity are two of the most critical features of a 
measure. Construct validity is concerned with whether the measure accurately represents 
the concept measured. Predictive validity is concerned with the power of the measure to 
predict future behavior or outcomes. 

5 There are many intended as well as unintended outcomes in a program. A 
descriptive approach focusing on the insider's perspective enables the e valuator to 
capture and document these outcomes. Ethnographic approaches are sensitive to 
outcomes that are not an explicit part of the program design, such as the leveling out 
process that occurs when a gifted child is grouped together with other gifted children and 
recognizes she is not the only bright child in the world. 

6 In one evaluation, student attendance was poor and teachers were inactive. 
Normally this would have resulted in a poor assessment. However, I probed and found 
that the sponsors did not support the program financially, preventing teachers from 
purchasing supplies required to conduct the most basic educational activities. Failure to 
probe would have resulted in a misevaluation. See Fetterman, 1981 for an example of 
blaming the victim. 

7 Triangulation is an instrumental tool to compare policy positions and actual 
program practice. In one evaluation, the policy stated that the "best teachers would be 
selected" based on superior teaching. Closer inspection revealed that teachers were 
selected according to seniority. The match proved less than optimal when the regular 
classroom teacher confronted the realities of educating challenging gifted and talented 
children. In this case, the interpretation and implementation of policy, rather than the 
policy itself, had a negative impact on the program. 

8 Participation is similar to but different from collaboration. Collaboration 
requires participation but takes one step beyond it. Collaboration requires the active 
participation of program participants in the study design, data collection, analysis, and 
report of findings. Collaboration and independence are not mutually exclusive. 
However, additional safeguards are warranted when a collaborative study is conducted. 

9 For example, a student with high IQ scores (intellectual potential) and low 
standardized achievement scores is probably an underachiever. 
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Notes (continued) 

10 A high IQ and high achievement test scores in mathematics probably should 
not be the driving criteria in the selection of students for a gifted and talented visual and 
performing arts program. 

11 In most psychometrically oriented approaches only the outcomes are 
revealed — process data are required to link the outcomes to specific program treatments. 

12 David Irvine, New York State Department of Education, Gifted Education 
Coordinator, has been an early, creative, and strong proponent of this approach. 

13 There are numerous problems with the use of standardized test scores with 
gifted and talented children, including regression toward the mean and ceiling effects. 
(See Thorndike & Hagen, 1969; Sattler, 1982 for additional discussion about these 
issues.) However, such test scores can be used to document change over time in student 
performance, particularly for underachievers and students from culturally diverse 
backgrounds. In addition, the use of appropriate standardized tests on which gifted 
students score around the mean might avoid some of the problems noted above. 

14 The works of Carter, Feldhusen, and Robinson also merit attention. 

15 There are many evaluation approaches that merit attention. For additional 
insight into different evaluation approaches, see Alkin et al, 1979; Campbell & Stanley, 
1966; Cronbach, 1980; Rossi & Freeman, 1989; Scriven, 1972; Stufflebeam, 1971; Stake, 
1975; Patton, 1990; Provus, 1969. 

16 Scriven refers to formative evaluation as evaluation aimed at the improvement 
of an ongoing enterprise (see Davis, Scriven, & Thomas, 1981). 

17 According to Davis, Scriven, and Thomas (1981), summative evaluation is 
conducted for an external client and its primary purpose is to report on the quality of the 
program for purposes other than improvement 
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Appendix A 

Nonjudgmenta! Orientation 

Some ethnographic concepts push the researcher or evaluator to explore in new 
directions, others ensure that the data are valid, still others simply prevent contamination 
of the data. A nonjudgmental orientation helps ethnographic evaluators on all three 
fronts. Most important, this concept prevents ethnographic evaluators from making 
inappropriate and unnecessary value judgments about what they observe. 

Ethnographers and ethnographic evaluators must attempt to view another culture 
without making value judgments about unfamiliar practices, but cannot be completely 
neutral. We are all products of our culture. We have personal beliefs, biases, and 
individual tastes. Socialization runs deep. However, the ethnographer and ethnographic 
evaluator can guard against the more obvious biases by making them explicit and by 
trying to view another culture's practices impartially. Ethnocentric behavior — the 
imposition of one culture's values and standards on another culture, with the assumption 
that one is superior to the other — is a fatal error in ethnography. 

The label ethnographic evaluator suggests a contradiction in terms to some 
scholars. How can an anthropologist be nonjudgmental and judgmental at the same time? 
The question provides a handle by means of which we can clarify the role of the 
ethnographic evaluator. 

First, it is a myth that anthropologists are completely nonjudgmental. The 
selection of a topic itself reflects built-in biases. The process of collecting data requires 
discrimination and judgment. Analysis and the manner in which findings are skillfully 
crafted and communicated reveal explicit and implicit biases. The aim, however, is to 
assume a nonjudgmental orientation toward different cultural practices. Ideally, value 
judgments are not made about marriage practices such as polygamy, gender-favored 
inheritance patterns, the lifestyle of a merchant or beggar, or personal hygiene practices. 
Both traditional ethnographers and ethnographic evaluators attempt to adopt this posture 
throughout a study and to make explicit their more conscious and obvious biases. A 
nonjudgmental orientation and an evaluative approach are not mutually exclusive. 
Evaluation simply represents another level of analysis. The evaluator can assess the 
functions and adaptations of a system, program, or policy without making a value 
judgment about the cultural practice per se. 

The major difference between the traditional ethnographer and the ethnographic 
evaluator is that the traditional ethnographer concludes the study with a description of the 
culture, whereas an ethnographic evaluator begins the evaluative segment of the study 
with a description of the culture. The ethnographic evaluator describes what is going on 
and then makes a qualitative leap beyond description to the explicit appraisal and 
assessment of the cultural system in terms of its own cultural norms. As an ethnographer 
and an ethnographic evaluator, I have found explicit assessment to be a more honest and 
useful approach to the study of human beings. (See Fetterman, 1984a; Fetterman & 
Pitman, 1986 for a more detailed discussion about the ethnographic evaluator.) 
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Appendix B 

Insider's or Ernie Perspective and Multiple Realities 

The emic perspective— the insider's or native's perspective of reality - is at the 
heart of most ethnographic research and ethnographic evaluation. The insider's 
perception of reality is instrumental to understanding and accurately describing situations 
and behaviors. Native perceptions may not conform to an "objective" reality, but they 
help the fieldworker understand why members of the social group do what they do. In 
contrast to a priori assumptions about how systems work from a simple, linear, logical 
perspective— which might be completely off-target— ethnography and ethnographic 
evaluation typically takes a phenomenologically oriented research approach. 

An emic perspective compels the recognition and acceptance of multiple realities. 
Documenting multiple perspectives of reality in a given study is crucial to understanding 
why people think and act the different ways they do. Differing perceptions of reality can 
be useful clues to individuals' religious, economic, or political status and can help a 
researcher understand maladaptive behavior patterns. (See Fetterman, 1989, for a 
detailed discussion of the emic perspective.) 
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Appendix C 
Triangulation 

Triangulation is one of the most basic analytical tools used in ethnographic 
research and ethnographic evaluation. Triangulation is at the heart of ethnographic 
validity. It involves testing one source of information against another to effectively strip 
away other plausible explanations for a hypothesis. Typically, the ethnographer or 
ethnographic evaluator compares one source of information with another, like a mini- 
experiment, to test the quality of the information collected (and the person sharing it), to 
better understand the part an actor plays in the human drama, and ultimately to put the 
whole thing into some sort of credible perspective. 

Triangulation can be used with any topic and in any setting and on any level of 
that setting. It always improves the quality of data and the accuracy of ethnographic 
findings. Triangulation can occur naturally in conversation as easily as in intensive 
investigatory work. However, the ethnographer or ethnographic evaluator must be 
prepared to identify it when it is happening in subtle contexts. (See Fetterman, 1989; 
Webb, Campbell, Schwartz, & Sechrest, 1966 for a detailed discussion about 
triangulation.) 
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Appendix D 

Participant Observation 

Participant observation characterizes most ethnographic research, as well as 
ethnographic evaluation, and is crucial to effective fieldwork. Participant observation 
combines participation in the lives of the people under study with maintenance of a 
professional distance that allows adequate observation and recording of data. 
Powdermaker's Stranger and Friend (1966) vividly depicts this delicate role. 

Participant observation is immersion in a culture. Ideally, the ethnographer lives 
and works in the community for six months to a year or more, learning the language and 
seeing patterns of behavior over time. Long-term residence helps the researcher 
internalize the basic beliefs, fears, hopes, and expectations of the people under study. 
The simple, ritualistic behaviors of going to the gifted and talented program seminar 
room or the lunch room teach how people use their time and space, and how they 
determine what is precious, sacred, and profane. 

The process may seem unsystematic; in the beginning, it is somewhat 
uncontrolled and haphazard. However, even in the early stages of fieldwork the 
ethnographer searches out experiences and events as they come to attention. Participant 
observation sets the stage for more refined techniques— including projective techniques 
and questionnaires— and becomes more refined itself as the fieldworker understands 
mere and more about the culture. Ideas and behaviors that were only a blur on entering 
the community take on a sharper focus. Participant observation can also help clarify the 
results of more refined instruments by providing a baseline of meaning and a way to re- 
enter the field to explore the context for those (often unexpected) results. 

Working with people— day in and day out— for long periods of time is what gives 
ethnographic research and ethnographic evaluation its validity and vitality. Given time, 
people forget their company behavior and fall back into familiar patterns of behavior. 
Ethnographic research in one's own culture may not require as much time to reach this 
point as ethnographic work in a foreign culture: Language and customs are familiar, and 
the researcher is already an insider in many respects. However, sometimes a familiar 
setting is too familiar, and the researcher takes events for granted, leaving important data 
unnoticed and unrecorded. 

In applied settings, participant observation is often noncontinuous, spread out 
over an extended time. For example, in one large-scale evaluation of a gifted program, I 
visited the program for only a few weeks every couple of months over a three-year 
period. The visits were intensive and included classroom observation, nonstop informal 
interviews, occasional substitute teaching, interaction with community members, and the 
use of various other research techniques, including long distance phone calls, dinner with 
student families, and time spent hanging out in the hallways and parking lot with 
students. 

Participant observation requires close long-term contact with the people under 
study. Often contract research budgets or time schedules do not allow long periods of 
stu dy — continuous or noncontinuous. In these situations, the researcher can apply 
ethnographic techniques to the study, but cannot conduct an ethnography. Similarly, 
observation without participation in other people's lives may involve ethnographic 
methods, but is not ethnography. Nonparticipant observation may be watching a school 
basketball game or a Board of Education meeting as part of data collection. Applying 
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ethnographic techniques and nonparticipant observation are acceptable forms of research, 
but labeling the research method correctly is important. 

The process may seem complicated, but a good ethnographer starts with the 
basics. Participant observation begins with the first question. Slowly but surely, the 
questions become more refined as the researcher learns what questions to ask and how to 
ask them. 

In any case, the acquisition of ethnographic knowledge and understanding is a 
cyclical process. It begins with a panoramic view of the community, closes in to a 
microscopic focus on details, and then pans out to the larger picture again — but this time 
with new insight into minute details. The focus narrows and broadens repeatedly as the 
fieldworker searches for breadth and depth of observation. Only by both penetrating the 
depth and skimming the surface can the ethnographer portray the cultural landscape in 
detail rich enough for others to comprehend and appreciate. 
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Appendix E 
Interviewing 

The interview is the ethnographer's and the ethnographic evaluator's most 
important data-gathering technique. Interviews explain and put into a larger context what 
ethnographers see and experience. They require verbal interaction, and language is the 
commodity of discourse. Words and expressions have different values in various 
cultures. People exchange these verbal commodities to communicate. The ethnographer 
quickly learns to savor the informant's every word for its cultural or subcultural 
connotations as well as its denotative meaning. General interview types include 
structured, semistructured, informal, and retrospective interviews. Although in practice 
these types overlap and blend, this discussion will artificially isolate interview types, 
strategies, and questions for purposes of description and discussion. Each interviewing 
approach has a role to play in soliciting information. However, the pros and cons of each 
interview type in data collection and analysis should be clear before employing these 
approaches in the field. (See Denzin, 1978; Goetz & LeCompte, 1984; Patton, 1990 for 
alternative approaches to classifying interviews. See also Bogdan & Biklen, 1982; 
Hammersley & Atkinson, 1983; Taylor & Bogdan, 1984; Werner & Schoepfle, 1987 for 
additional discussion about interviewing techniques.) 

Formally structured and semistructured interviews are verbal approximations of a 
questionnaire with explicit research goals. These interviews generally serve comparative 
and representative purposes — comparing responses and putting them in the context of 
common group beliefs and themes. The fieldworker can use a structured interview at any 
time in the study. For example, a list of questions about the educational background of 
the teachers in a school under study is useful in securing comparative baseline data about 
the teachers 1 qualifications and experience. It can also be a nonthreatening icebreaker. 
At the beginning stages of a study, however, structured interviews tend to shape 
responses to conform to the researcher's conception of how the world works. These 
interviews are, therefore, most useful at the middle and end stages of a study to collect 
data about a specific question or hypothesis. A structured or semistructured interview is 
most valuable when the fieldworker comprehends the fundamentals of a community from 
the "insider's" perspective. At this point, questions are more likely to conform to the 
native perception of reality than to the researcher's. 

Informal interviews are the most common in ethnographic work. They seem to be 
casual conversations, but where structured interviews have an explicit agenda, informal 
interviews have a specific but implicit research agenda. The researcher uses informal 
approaches to discover the categories of meaning in a culture. Informal interviews are 
useful throughout an ethnographic study to discover what people think and how one 
person's perceptions compare with another's. Such comparisons help identify shared 
values in the community — values that inform behavior. Informal interviews are also 
useful in establishing and maintaining a healthy rapport 

Informal interviews seem to be the easiest to conduct. They do not involve any 
specific types or order of questions, and can progress much as a conversation does, 
following the turns of the participant's or the questioner's interests. However, these 
interviews are probably the most difficult to conduct appropriately and productively. 
Issues of ethics and control emerge from every informal interview. How does the 
fieldworker establish and maintain a natural situation while attempting to learn about 
another person's life in a relatively systematic fashion? How can a completely open form, 
ripe for discovery, balance with an implicitly shaped structure designed to explore 
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specific issues and concerns? Finally, when is the time to take advantage of a golden 
opportunity, and when is it best not to pry further? Done well, informal interviewing 
feels like natural dialogue but answers the fieldworker's often unasked questions. 

Informal interviews should be user friendly. In other words, they should be 
transparent to the participant after a short period of time. An informal interview is 
different from a conversation, but it typically merges with one, forming a mixture of 
conversation and embedded questions. The questions typically emerge from the 
conversation. In some cases, they are serendipitous and result from comments by the 
participant. In most cases, the ethnographer has a series of questions to ask the 
participant and will wait for the most appropriate time to ask them during the 
conversation (if possible). 

Informal interviews offer the most natural situations or formats for data collection 
and analysis. Unfortunately, some degree of contamination is always present. However 
skillful the interviewer, certain questions will impose an artificiality. An experienced 
interviewer, however, learns how to begin with nonthreatening questions deeply 
embedded in conversation before posing highly personal and potentially threatening 
questions and to develop a healthy rapport before introducing sensitive topics. Sensitivity 
to timing and to the participant's tone is critical in interviewing — informal or otherwise. 
An ethnographer must learn to be attentive to a person's shifts in tone because these 
changes are important cues to attitudes and feelings. (See Fetterman, 1983 for a 
discussion of the ethical hazards ethnographers (and specifically ethnographic evaluators) 
face in the field.) 

The chance to exploit a vulnerable individual to secure invaluable data may be 
tempting. In fact, it may be a rare opportunity to explore an individual's innermost 
secrets. However, beyond the obvious ethical considerations, the cost of exploiting an 
individual is too high, and the ethnographer must wait for another opportunity to come 
along or create one. One benefit of spending long periods of time at a site is that other, 
more propitious opportunities usually come along. Over sensitivity, however, can 
paralyze an ethnographer, placing unnecessary obstacles in the way of data collection and 
analysis. 

A multitude of significant nonthreatening questions can elicit the information the 
fieldworker seeks and create many golden moments in which to ask questions naturally, 
as part of the general flow of conversation. Planning and executing properly placed 
questions is the essence of good ethnography, ensuring the quality of the data and 
maintaining the participant's rights to privacy. 

Retrospective interviews can be structured, semistructured, or informal. The 
ethnographer and ethnographic evaluator uses retrospective interviews to reconstruct the 
past, asking informants to recall personal historical information. This type of interview 
does not elicit the most accurate data. People forget or filter past events. In some cases, 
retrospective interviews are the only way to gather information about the past. In 
situations where the ethnographer already has an accurate understanding of the historical 
facts, a retrospective interview provides useful information about the individual. The 
manner in which individuals shape the past highlights their values and reveals the 
configuration of their world view. 

Ethnographers and ethnographic evaluators use interviews to help classify and 
organize an individual's perception of reality. All interviews share some generic kinds of 
questions. The most common types are survey or grand tour, detail or specific, open- 
ended or closed questions. Survey questions help identify significant topics to explore. 
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Specific and detailed questions explore these topics in more detail. They determine 
similarities and differences in the ways people see the world. Open-ended and closed- 
questions help the ethnographer and ethnographic evaluator discover and confirm the 
participant's experiences and perceptions. (See Fetterman, 1989 for a more detailed 
discussion of interviewing approaches and techniques.) 
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